Part of my everyday work is related to podcasts. With podcasts, what really takes time is often not the content itself, but post-production, especially audio editing.
Editing audio in the traditional way takes a very, very, very long time. But this is the AI era now, and efficiency has almost become an assumption.
That is why I keep noticing editing tools that claim to be ‘more efficient,’ sometimes deliberately and sometimes without meaning to.
I came across one idea early on: document-based editing. Products like it appeared in China. The basic logic was to turn audio into text, then edit the audio as if editing a document. Looking back now, that product has almost disappeared. Even later products I tried still have not solved the problems it left behind.
This shows that document editing does not work in a Chinese context.
Later I tried Descript. For a while, it was probably a star product that came up again and again in podcasting.
But I found that its core logic for audio editing was still document editing. People in China had already run into this wall long ago. Clearly, it was not specially designed for Chinese situations, even if it supports Chinese speech recognition.
It confirmed something further: document editing has a ceiling as a path for audio editing.
Audio carries lots of information that is not text: breaths, background noise, ambient sound. Whether those sounds should be removed requires choices and judgment, and document-editing tools struggle to make those choices for people.
If a tool cannot recognize those parts, I still have to listen through the audio from beginning to end and check them one by one. Then the so-called efficiency gain makes no sense; it is even more troublesome.
There is also a deeper reason: audio signals and text are very far apart. Text is a set of discrete symbols; audio is a continuous signal.
Deleting one word in a document is only deleting a symbol. In audio, it means making two hard cuts on a timeline and forcing the parts before and after together. Without fade-ins and fade-outs at the join — which also need different treatment in different situations — you can easily get sharp, unpleasant sounds.
There are even subtler cases.
The position of words on a timeline, especially in Chinese, does not always match a good place to cut audio. The sound of one character often overlaps with the syllables before and after it. If you cut directly at word boundaries, it is easy to remove the end of the previous character or the beginning of the next one, making it uncomfortable to hear.
Neither of these situations can be solved by transcription accuracy. At their core, they are problems of hearing.
So document editing may be an entrance, but it is not the end point of audio editing. What should an audio-editing tool for heavy users really become?
If the goal is to reduce editing time, there seem to be only a few paths. Either the tool tries to judge directly for the person, or it learns that person's way of judging.
In other words, the editor gradually moves from being an executor who repeats operations to being someone mainly responsible for judgment. Then the question becomes concrete: how does a tool learn an editor's style?
In a traditional audio-editing tool such as Audition, I can imagine a fairly natural process.
At first, the editor works as usual: creates a project, imports audio, and begins in their own way. At the same time, a shadow editing assistant runs in the background, quietly watching the entire editing process.
After watching for a while, when it feels it roughly understands this editor's style, it appears without interrupting and says: I am ready. I can try editing a small section.
If the editor agrees, the shadow AI shows the small section it has edited. Once the editor listens, they naturally judge whether it is okay.
If it is okay, let it continue. If not, the editor will surely make manual changes.
Those changes are exactly the shadow AI's most important learning material.
It can compare what it originally did with the editor's revised result. The difference itself is part of editing style.
As long as this process continues, every human change feeds back into what the shadow AI learns. Over time, it gradually learns that editor's style.
What the shadow AI learns is not only how to edit one piece of audio, but a way of judging that can be saved.
If that judgment could eventually exist as a local file — a style file, for example — it would no longer only be part of the tool. It would be an asset, an asset belonging to the editor.
The editor could use it alone, share it, or even commercialize it. For people who want to learn a particular editor's style, this would be much more direct than analyzing finished work afterward.
I often run into this myself. For example, I may love how an editor presents their work — this is a video example, but it applies to audio too, such as people who are especially good at pacing and cuts — without knowing how they made those effects. Then I can only screenshot things or give the result to AI to analyze. For audio, AI is still very bad at understanding and analyzing pacing and cuts. It is a huge blank area 🤔
But that kind of analysis is worlds apart from an editor's real operations and judgments. I end up learning something neither here nor there. There is a benefit too: taking a little of another person's style and making your own. Okay, that is not the point.
If an editor can turn their judgments into a style file, it becomes an important decision entry point for learners. It lets someone directly work in a certain way.
Recently I came to understand a term deeply: continual learning.
For a shadow editing assistant, memory is dynamic. Every manual operation, modification, and tiny adjustment from the editor is necessary memory.
Its learning happens continuously inside the editing process.
Audio editing will always have edge cases. No system can cover every possibility at once. That also means a shadow AI cannot learn one fixed upper limit at some point. It can only keep learning and consolidating through continued use.
At the same time, it needs to gradually learn a judgment: when it faces a particular episode, which past experience should it use, and which should it set aside for now? These choices in editing slowly become style too. That style is not decided by AI alone; it is constantly checked through the editor's feedback. When the shadow AI finishes a section, the editor naturally responds. Lots of feedback means the learning is still not aligned enough. Little feedback means most judgments are aligned and only details need adjusting or adding.
It is like how people learn before major exams. When you get many exercises wrong, you need to return to the basics. When you only get a few wrong, you have usually understood most of it. If this continual learning lasts for a long time, the shadow AI can complete a considerable share of editing in most cases, while the editor appears more as a judge and decision-maker. Wouldn't that save time and energy for more important things?
Of course, this is still only a relatively ideal imagination. It is hard to say what problems would appear in real implementation.
But one thing is clear to me: this kind of learning should happen locally.
Both the shadow AI's learning process and its final style file are better as local data.
Tools like Audition already support local saving. In reality, most heavy editors also prefer to keep materials and projects locally.
If editing data, judgment data, and style assets all stay local, editors can have full control over their work assets. They are not automatically uploaded to the cloud or turned into training material for a platform without clear agreement. Under that condition, a shadow AI is more like an extension of the editor, rather than part of a platform. It also fits the current move toward local privacy.
From the point of view of use cases, this idea makes sense for audio editing.
There are still many podcasts in China that are mainly audio. Many people listen to podcasts simply to listen, not to watch video. ‘Video podcast’ is a slightly strange phrase to me, even though I have said it countless times. 🤔
Audio editing also does not involve matching picture and sound. Its data dimensions are more limited and its chain of judgment is simpler. That is why my imagination stays mainly at audio editing for now. Video editing needs a different design and is another, more complex path.
Still, audio can go from Premiere Pro to Audition for processing, then back to Premiere Pro. Does that make the possibility of edited audio synchronizing with its matching images much greater? What step is missing? The thinking behind that is very interesting!!
At least for audio editing, this idea seems very reasonable to me and worth trying for someone with the ability. In fact, the relevant platforms are better placed to be that someone, especially companies in audio and video. So which company can actually make this happen?
I tried. I crashed into wall after wall... and still could not make it work ahhhhh!
Oh, people with editing experience will definitely say: why are you so attached to Audition? Use Jianying. It can edit automatically. There are also lots of automated video-editing tools, and you could export the audio at the end...
In this article, I wrote about my experience using Jianying and CapCut. Current automated editing tools mostly serve audiovisual effects that are fast-paced, emotionally intense, or good at grabbing attention. That does not fit what I expect and need from podcast production or presentation. (My standards are really high... :()
Still, I have to admit that listeners do not seem to care much about sound quality.
If listeners' ears are sensitive enough, they can definitely hear which major podcasts use similar automated tools to help edit. When I hear it, I wonder: why can everyone seem to accept it?
Sorry, I am the kind of person who immediately runs far away... :)
I get a similar feeling when I face other people's creative work. I even found a friend complaining about this in Moments. At the time I thought: wow, bro! Let me shake your hand from afar! I have been suffering and complaining about this for so, so, so long!!
Are people indifferent to writing that obviously carries AI traces, or have they already accepted this way of receiving information by default? (ChatGPT should be @'d here hahahaha — if we can correct the ‘AI feeling’ through prompts, it must be possible at the system level too. But... why is it not done? That is another question. 🤔)
But I need a disclaimer: I am not against AI-assisted creation. What I object to is being unable to edit AI's writing into your own style — or at least into writing that feels normal to read. Okay, I should work harder too: give spoken expression more chances and spend less time typing. :)
(Echoing continual learning and dynamic learning!)