Running Private ASR on Ascend 910B: Setup, Operator Adaptation and Throughput

This post walks through landing a Chinese speech-to-text engine on Huawei Ascend NPUs in a private environment — environment versions, model conversion, deployment commands, pitfalls and reference performance figures.

On the numbers: figures marked "spec" come from published product specifications. The performance table gives reference magnitudes under typical conditions; actual values vary with audio conditions, model size and concurrency strategy. Always measure on your own workload.

Why It Has to Run on Domestic Hardware

In government, finance, and defense, systems must be not only "data on-premises" but also "technologically self-reliant." That means shifting the underlying compute from imported GPUs to Ascend, Cambricon, and Hygon domestic chips, and the OS to Kylin, UnionTech, and other domestic systems.

In engineering terms this becomes three hard constraints:

• Data residency — meeting audio contains internal discussion, and many organizations explicitly forbid audio from leaving the intranet
• Xinchuang procurement — imported GPUs may not appear on the approved purchase list at all
• Air-gapped operation — some sites have no external connectivity, so model weights must be packaged into the environment up front

Core point: Private deployment answers "where the data is"; localization answers "who controls the technology." Together they converge on a single task — get ASR running on domestic NPUs, and running fast enough.

Supported Accelerators

Built on NEU NLP (Northeastern University) research, the VoiVision engine natively supports mainstream accelerators:

Type	Representatives	Use
NVIDIA	T4 / L4 / A10 / A100	General, high performance
Ascend	310P / 910B	Localized, domestic servers
Cambricon	MLU370 / MLU590	Localized, domestic servers
Hygon DCU	DCU series	Domestic x86 path
CPU only	x86 / ARM	Reuse existing, low concurrency

Ascend currently has the most complete ecosystem among domestic accelerators (CANN + MindSpore + model zoo). The 910B targets training and high-throughput inference, while the 310P targets edge and low-power use. For sustained concurrent meeting transcription, a common pattern is 910B for central inference and 310P for edge nodes.

Environment Setup: Version Alignment Is the Biggest Trap

The pain in the Ascend ecosystem is not compute, it is the version matrix. Driver, firmware, CANN, framework and model format — any one of the five being out of sync produces baffling failures.

A combination we commonly see stable:

Component	Version	Note
Driver / firmware	Tightly bound to CANN	Check the official compatibility table before upgrading
CANN	8.x series	Operator library and model conversion tools live here
Container	Ascend Docker Runtime	Required — plain Docker cannot mount the NPU
Inference framework	ONNX Runtime + ACL	OM models work fine without MindSpore
OS	openEuler / Kylin V10 / UnionTech UOS	Common in Xinchuang environments

Step one is always confirming the NPU is visible:

# List NPU devices and status
npu-smi info

# Expect a device list with 910B entries in a healthy state
# If the card is not visible here, nothing downstream matters — fix the driver first

Pitfall: on some Kylin V10 systems, npu-smi info reports a dmp daemon error right after driver installation. The cause turned out to be a conflict between the system's own dkms version and the driver installer. The fix is to remove the system dkms and reinstall using the version bundled with the driver package. Distribution-level adaptation issues like this are rarely documented and are among the most common on-site blockers.

Model Conversion: ONNX to OM

ASR models are typically trained in PyTorch, exported to ONNX, then converted to Ascend's OM format.

Three Things to Watch When Exporting ONNX

torch.onnx.export(
    model, dummy_input,
    "asr.onnx",
    opset_version=14,          # Ascend supports lower opsets better — avoid the newest
    do_constant_folding=True,
    input_names=["audio", "audio_len"],
    output_names=["logits"],
    dynamic_axes={              # critical: audio length is variable
        "audio": {0: "batch", 1: "time"},
        "logits": {0: "batch", 1: "time"},
    },
)

Three places where this commonly goes wrong:

1. Opset version — newer is not better. High opset versions hit unsupported operators more often during conversion; 14 is a well-tested choice.
2. Dynamic axes — audio length is inherently variable, so dynamic_axes must be declared, otherwise you are limited to fixed-length input and the result is unusable in production.
3. Constant folding — do_constant_folding=True noticeably shrinks the graph and improves conversion success.

Conversion with ATC

atc --model=asr.onnx \
    --framework=5 \
    --output=asr_om \
    --input_format=ND \
    --input_shape="audio:-1,-1;audio_len:-1" \
    --dynamic_dims="1,16000;1,32000;1,48000" \
    --soc_version=Ascend910B \
    --precision_mode=allow_fp16 \
    --log=error

What the parameters actually mean:

• --soc_version — must match the physical hardware exactly. Specifying 910A on a 910B fails at load time with a vague error.
• --dynamic_dims — dynamic shape profiles. For audio, profile by duration; too many profiles increases compile time, too few triggers frequent recompilation and latency jitter.
• --precision_mode — allow_fp16 typically costs negligible accuracy on ASR while improving throughput clearly.

Should You Quantize?

Precision	Throughput	Accuracy
FP16	Baseline	Baseline
INT8	Clear gain	Noticeable cost

Verdict: keep FP16 where accuracy is critical (medical, legal meetings); INT8 is the better trade where error tolerance is higher, such as internal stand-ups and training recordings.

Deployment: Containers Are the Only Sensible Option

Bare-metal installs are nearly unmaintainable on Xinchuang sites. We deliver containers:

docker run -it --name asr_server \
  --device /dev/davinci0 \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm \
  --device /dev/hisi_hdc \
  -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
  -v /data/models:/models \
  -p 8000:8000 \
  asr-server:910b \
  ./asr_server --model /models/asr_om --port 8000

• Runs on bare metal, government cloud, and localized environments;
• Integrates with OA and meeting systems via standard RESTful / WebSocket APIs;
• Scales elastically by concurrency.

Note those --device entries: missing any one of them produces an ACL initialization failure, and the error will not tell you a device is missing. This is the most common stumbling block for newcomers.

Also — do not invent a private protocol. Use standard REST (file transcription) and WebSocket (real-time streaming), or integrating with the customer's OA and meeting systems becomes painful.

If you already have servers, deploy the software-only engine to get private transcription on domestic accelerators without buying hardware.

Reference Performance

Test conditions: Ascend 910B, FP16, Chinese meeting audio, 16 kHz mono.

Metric	Reference	Note
Real-time factor (RTF)	~0.1 order	one hour of audio in a few minutes
Concurrent streams per card	Tens	Pure transcription, sustained meeting concurrency
First-token latency	Sub-second	Streaming mode; typically under 1s in practice
Chinese transcription accuracy	≥98%	Standard Mandarin meeting scenarios (spec)
Dialect coverage	22	Auto-detected (spec)

⚠️ The concurrency figure above is transcription only (single ASR card, no speaker separation or minute structuring). Adding speaker separation plus topic/action-item extraction lowers per-card capacity significantly — the appliance is rated at 10 concurrent meetings, which refers to the full pipeline (transcription + separation + structuring); the cluster form is rated at 50+ streams. Always ask which layer a vendor is quoting — the three differ by multiples.

Six Pitfalls From Real Deployments

This is the most valuable part of the post. Every one of these was hit on site.

1. Dynamic shapes causing constant recompilation Without --dynamic_dims, every new audio length triggered a graph recompile and latency spiked to seconds. Profiling by duration brought it back under a second.

2. Long audio exhausting memory Feeding a two-hour meeting recording straight into the model OOMs the device. You need chunked incremental inference — segment by VAD and decode segment by segment rather than loading the whole file.

3. Unsupported operators failing silently A LayerNorm variant had no Ascend counterpart. ATC raised no error during conversion, and the model output all zeros at runtime. Always run an equivalence check after conversion: run the same audio through ONNX Runtime (CPU) and OM (NPU) and compare outputs.

4. Context leaks under multi-threaded concurrency An early version created a separate ACL context per request and exhausted contexts under load. Context pooling fixed it.

5. Models must be reconverted after driver upgrades After a major CANN upgrade, older OM models may fail to load or lose performance. Put model conversion in CI rather than converting by hand once.

6. Offline packaging for air-gapped sites In physically isolated environments, CANN dependencies, drivers, model weights, fonts and certificates all have to be packaged in advance. We once arrived on site to find a missing CJK font, and every transcript came back as tofu boxes. Rehearse the full deployment on an equivalent environment before shipping the offline bundle.

When Not to Choose Domestic Accelerators

Being straight with you. Skip it if:

• There is no Xinchuang or localization mandate → the GPU ecosystem is far more mature, do not make work for yourself
• Nobody on the team knows CANN → operations cost will far exceed whatever the hardware saves
• Concurrency is single digits → plain CPU is enough, skip the accelerator

Domestic accelerators are the right call when: there is a clear compliance mandate + sustained concurrency pressure + engineering capacity to back it up.

Sizing Advice

• New localized cluster: VV05 / VV10+ Ascend or Cambricon accelerator;
• Reuse existing x86: engine on CPU or Hygon DCU;
• Mixed environment: unify NVIDIA and domestic accelerators, schedule by load.

If you are weighing private against cloud, see the on-premise ASR selection guide.

See the private deployment guide and MLPS Level 3 compliance.