War Story: Implementing solution for 4G/5G voice call verification on AOSP quickly

1. Speaker：Wig Cheng Date：2023.11.11
1

2. The Title Me
Wig Cheng
• Focus on Android/Linux BSP
• Open source contributor
• Android: Google Kernel (GKI) upstream
• Linux Kernel upstream
• Denx U-boot upstream
• ROS (Robotic OS framework)
• IEI World Group – Android OS Engineering Manager

3. CONTENTS
01
War beginning
Introduce different cases for this situation
02
First Step – Confirm HW design
03
Quickly setup phone call subsystem
04
New Challenge
3
05
In the end?

4. War beginning
Android applications, motivation, and some mobile phone call cases
1
4

5. The Title
Classic
Smart Phone
Tablet
Smart watch
Android applications
Enterprise
Industrial HMI
Fitness
POS (Point-Of-Sell)
AOSP
5
Focus on here today

6. The Title
Use case
Transportation
Bus driver’s HMI
(Human Machine Interface)
Motivation
HDMI
Speaker, Microphone
Headset
Focus on audio function:
1. HDMI audio output – for passenger
2. Speaker, microphone – for bus driver
3. Headset – for bus driver
4. Phone call via modem – for bus driver
6

7. The Title Mobile phone call – Normal case I
Open Source project: PinePhone Pro
OS: Android 12
Linux Kernel
HAL driver
Framework
Phone APK
Audioflinger service
Audiopolicy service
tinyHAL
Audio configuration
RT5640
XML file
7

8. The Title Mobile phone call – Normal case II
Linux Kernel
Framework
Phone APK
Audioflinger service
Audiopolicy service
Modem SOC Bluetooth
HAL driver
tinyHAL
Audio configuration
XML file
Open Source project: PinePhone
OS: Android 12
8

9. The Title PinePhone – XML example
9
<audiohal>
<mixer name="PinePhone">
<init>
<!-- AIF1 Digital input (SoC) -->
…
<!-- AIF2 Digital input (modem) -->
<!-- Default modem playback volume -->
<ctl name="AIF2 DAC Playback Volume" val="160"/>
<!-- AIF2 to DAC mono playback -->
<ctl name="AIF2 DAC Stereo Playback Route" val="Sum Mono"/>
<!-- From modem to DAC switch -->
<ctl name="AIF2 Digital DAC Playback Switch" val="1"/>
<!-- AIF2 Digital output (modem) -->
<!-- Default ADC to AIF2 volume -->
<ctl name="AIF2 ADC Capture Volume" val="160"/>
<!-- ADC to AIF2 mono playback route -->
<ctl name="AIF2 ADC Stereo Capture Route" val="Mix Mono"/>
<!-- Enable AIF1 source to modem switch -->
<ctl name="AIF2 ADC Mixer AIF1 DA0 Capture Switch" val="1"/>
<!-- From ADC to modem switch -->
<ctl name="AIF2 ADC Mixer ADC Capture Switch" val="1"/>
…
</mixer>
<!-- AUDIO_DEVICE_OUT_SPEAKER -->
<device name="speaker">
<path name="on">
<ctl name="Line Out Playback Switch" val="1"/>
</path>
<path name="off">
<ctl name="Line Out Playback Switch" val="0"/>
</path>
</device>
<stream type="pcm" dir="out" cardname="PinePhone" device="0"
rate="48000">
</stream>
<stream type="pcm" dir="in" cardname="PinePhone" device="0"
rate="48000">
</stream>
</audiohal>

10. The Title Mobile phone call – current case
ARM
Cortex-A72
HDMI
Audio codec
Modem audio
Linux Kernel
Framework
Phone APK
Audioflinger service
Audiopolicy service
Audio codec
Ideal case
HDMI Modem
Current project
OS: Android 9
HAL driver
tinyHAL with
Audio configuration
I2S
I2S
PCM
10
No xml file, setting
inside the HAL

11. The Title
Mobile phone call – current case (cont.d)
Linux Kernel
Framework
Phone APK
Audioflinger service
Audiopolicy service
Audio codec
Ideal case
HDMI Modem
HAL driver
Real status:
• Issue
• Only hear noise when do a phone calling
• Customer’s requirement
• HDMI as primary audio card in Android OS
• Enable G-sensor function
• Fix WiFi crash randomly issue
• Enable phone call function (partially works)
• Note that LTE networking is works (lucky!)
• Only three weeks work left
11
tinyHAL with
Audio configuration

12. The Title
Mobile phone call – current case (cont.d)
12

13. First Step
Confirm HW design
2
13

14. The Title Mobile phone call – current case
Linux ALSA userland
• /dev/snd/
• ALSA library
• Utilities
• arecord
• aplay
• amixer
• alsamixer
Android TinyALSA userland
• /dev/snd/
• tinyALSA library
• Utilities
• tinycap
• tinyplay
• tinymix
14
CPU DAI
Codec DAI
ALSA driver = Advanced Linux Sound Architecture
Machine driver = bridge between CPU side and codec side
Codec driver = codec registers and clock setting
ASOC = Advanced Linux Sound Architecture for SoCs
DAI = Digital Audio Interface

15. The Title Device tree modification
+ modem_soundcard: modem_soundcard
+ {
+ status = "okay";
+ compatible = "simple-audio-card";
+ simple-audio-card,name = " modem_soundcard";
+ simple-audio-card,format = "dsp_a";
+ simple-audio-card,mclk-fs = <256>;
+ simple-audio-card,bitclock-master = <&dailink0_master>;
+ simple-audio-card,frame-master = <&dailink0_master>;
+
+ simple-audio-card,cpu {
+ sound-dai = <&i2s1>;
+ };
+
+ dailink0_master: simple-audio-card,codec {
+ sound-dai = <&dummy_codec>;
+ };
+ };
15
+ dummy_codec: dummy-codec
+ {
+ status = "okay";
+ compatible = " gtm601";
+ #sound-dai-cells = <0>;
+ clocks = <&cru SCLK_I2S_8CH_OUT>;
+ clock-names = "mclk";
+ };
+&i2s1 {
+ #sound-dai-cells = <0>;
+ status = "okay";
+};

16. The Title Dummy codec driver?
16
Refer from mainline driver
• <kernel source>/sound/soc/codecs/gtm601.c
static struct snd_soc_dai_driver gtm601_dai = {
.name = "gtm601",
.playback = {
.stream_name = "Playback",
.channels_min = 2,
.channels_max = 2,
.rates = SNDRV_PCM_RATE_8000,
.formats = SNDRV_PCM_FMTBIT_S16_LE,
},
.capture = {
.stream_name = "Capture",
.channels_min = 2,
.channels_max = 2,
.rates = SNDRV_PCM_RATE_8000,
.formats = SNDRV_PCM_FMTBIT_S16_LE,
},
};
Modem Audio Spec.
Data length = 16-bit
Sample rate = 8kHz
Channel = 2
BCLK = 256KHz
LRCLK = 8KHz = 16*2*2 = 64fs

17. The Title Mobile phone call – current case
Test Modem audio card
• tinycap
• tinyplay
Bclk = channel * sample rate * data length
= 2 x 8000 x 16 = 256000Hz
17
Device
Modem audio card
# cat /proc/asound/cards
0 [max98090codec ]: max98090codec - max98090codec
max98090codec
1 [modem_soundcard]: modem_soundcard - modem_soundcard
modem_soundcard
2 [rkhdmidpsound ]: rk-hdmi-dp-soun - rk-hdmi-dp-sound
rk-hdmi-dp-sound
# ls /dev/snd/
controlC0 controlC1 controlC2 pcmC0D0c pcmC0D0p pcmC1D0c
pcmC1D0p pcmC2D0p
tinycap test.wav -D 1 -r 8000 -c 2
tinyplay test.wav –D 1

18. Quickly setup phone
call subsystem
3
18

19. The Title How to do?
Current weapons
• HAL? (Standard way)
• Primary: HDMI, Secondary: Codec
• Modem doesn’t works
• Worry about side effect if force adding
• tinycap
• tinyplay
• Linux tool: ModemManager (mmcli)
• mm-radio HAL driver
19
Device
codec modem
OUT IN OUT IN
tinycap
tinyplay
test.wav

20. The Title How to do?
Current weapons
• HAL? (Standard way)
• Primary: HDMI, Secondary: Codec
• Modem doesn’t works
• Worry about side effect if force adding
• tinycap
• tinyplay
• Linux tool: ModemManager (mmcli)
• mm-radio HAL driver
• New idea!!?
20
Device
codec modem
OUT IN OUT IN
pcm channel
pcm channel
Write a new daemon in <AOSP source>/external/tinyalsa/

21. The Title How to do?
21
Phone app
Android Framework
ART (Android Runtime)
Audio Service
Audio HAL layer
(HDMI, codec)
Native Library layer
JAVA API framework
Android OS
Linux Kernel Layer
Audio subsystem
Modem sound card codec sound card HDMI sound card
Hardware (including MODEM)

22. The Title Difficult Point
22
• How to get phone call status
• How to check audio card status
# dumpsys telephony.registry | grep mCallState | tr –d ‘ ’
mCallState=0 (No any phone call, idle status)
mCallState=1 (Calling in)
mCallState=2 (Incalling)
open audio card using tinyalsa in different situation
Scenario 1:
Scenario 2:
Scenario 3:
Device Phone
Device Phone
Device Phone
mCallState=0
Codec and modem card are free
mCallState=2
Codec and modem card are free
mCallState=from 1 to 2
1: codec was be occupied by audio service
Du~Du~
Ring~~

23. The Title Create new functions!
23
Other applications using this way
• Loopback mode
• AI smart feedback
• Voice Transfer
• Multiple sound cards bridge
• HDMI IN
• Audio capture card
• Patent pending on TW, CN, US
Device
codec modem
OUT IN OUT IN
AI inference
and feedback
Device
Card 1 Card 2 Card 3 Card N
Voice transfer

24. The Title Testing and fix issue
24
Click noise issue
• Observation using Audacity tool (FOSS tool)
• Remove click noise from subsystem
Trade-off:
• Fully filtering click noise
• Worse latency in data transfer of PCM channel
Brute Force Attack

25. New Challenge
4
25

26. The Title Field testing
26
Customer side
• Only heard noise when phone call
• Checking software part: Same
• Checking hardware revision: Same
• Checking SIM card provider: Same
• Checking modem module: ???

27. The Title Game time: Look for difference
27

28. The Title Game time: Look for difference
28
Vendor: PCB S6B already EOL!

29. The Title Experiments
29
List difference
PCB S6 (NEW with noise) PCB S6B (OLD) Vendor FAE feedback
PCM data stereo channels stereo channels Both the same
AEC via AT command Failed No need No comments
add 33pF capacitor to
PCM clock pin
loopback works perfectly
Record to wav: still has issue
Loopback: starting has
big noise…
No comments
Add 33pF capacitor to I2S
of codec side
Failed No need No comments
Catch modem log using
diag tool (Linux based)
Can be used Can be used Need fine tune codec side
after analysis…
Modem
Audio card
CPU Audio codec
33pF
pcm_bclk

30. The Title PCM format analysis
30
• PCM_A
• PCM_B

31. The Title PCM format analysis (cont.d)
31
• Modem format
Should be PCM_A, but it works when change
to PCM_B in device tree.
+ modem_soundcard: modem_soundcard
+ {
+ status = "okay";
+ compatible = "simple-audio-card";
+ simple-audio-card,name = " modem_soundcard";
+ simple-audio-card,format = "dsp_b";
+ simple-audio-card,mclk-fs = <256>;
+ simple-audio-card,bitclock-master = <&dailink0_master>;
+ simple-audio-card,frame-master = <&dailink0_master>;
+
+ simple-audio-card,cpu {
+ sound-dai = <&i2s1>;
+ };
+
+ dailink0_master: simple-audio-card,codec {
+ sound-dai = <&dummy_codec>;
+ };
+ };
Refer from
<kernel source>/Documentation/devicetree/bindings/sound/simple-card.txt

32. The Title Test results
32
Changed list
• Change PCM format to dsp_b
• Add 33pF capacitor to PCM_BCLK pin
Results:
1. New modem works perfectly without any click noise.
2. So remove the algorithm click noise filter from last chapter.
3. First order already mass production and shipping!

33. In the end?
5
33

34. The Title Conclusion
34
New phone call subsystem
• Pros
• Easy to bridge any two sound cards
• Independent daemon, low latency
• Independent PCM channel with any audio algorithms
• tinyALSA based, so it can easy to move NUTTX OS
• Cons
• NOT match AOSP standard rule
1. CTS/VTS will be NG
2. SELINUX issue
But it’s not a problem if the user doesn’t matter

35. The Title Conclusion (cont.d)
35
Face a large scale problem in system integration with limited
schedule, you have to...
• Calm down
• Don’t ignore small things
• Compare with other open source projects
• Don’t fully trust vendor’s comments
• Don’t fully trust hardware design
• Try to find a way to validate the function works first

36. The Title The war still going on
36
Echo issue
• Mechanism way
• PCM algorithm way
Device
codec modem
OUT IN OUT IN
pcm channel
pcm channel
Hello!
Hello!

37. The Title Reference
[1] Porting custom audio codec in Linux BSP
[2] Google - Android audio architecture
[3] Audacity official website (freeware)
[4] I2S/PCM 接口介绍
[5] CSDN - android tinyalsa 理解
[6] Linux Kernel ORG. – Machine driver: Simple audio card README file
[7] Google - AOSP architecture
[8] Meme梗圖產生器
37

38. Speaker：Wig Cheng Date：2023.11.11
38