macOS v1.3.4b now available on TestFlight

This is a really exciting update because it gets Ally much closer to the vision of being “local AI that doesn’t suck.” The focus of this update is building a strong performance foundation and adding early agentic capabilities.

New: Notch File Drop

Ally is a productivity app focused on “quick capture” which is basically the idea that getting things out of your head and into Ally should be as frictionless as possible.

To that end, we’ve added Notch File Drop for quick and easy processing. Drag and drop PDF, audio, Markdown, and text files into the Notch button to automatically convert them to Markdown and process them into Ally’s “memory” (Context engine).

PDFs are parsed into Markdown, audio files are instantly transcribed, and a new Node is added to Ally.

This immediately begins a background process that digests the document for RAG, extracts knowledge graph entities and relationships, summarizes it so it’s easily found in the future, and more, making all that data accessible to Ally.

Document types supported:

• .pdf
• .md
• .txt

Audio file types supported:

• .m4a
• .mp3
• .wav
• .aiff
• .aac
• .caf

So Notch File Drop is another way to quickly add knowledge to your Ally. Each PDF you add, each web clipping, each meeting transcript adds to Ally’s “memory” which has also gotten massive improvements in this update…

Improved: Massive context engine improvements

A key product focus for Ally is its sophisticated context engine for local AI chat. The goal is for Ally to excel at “needle in a haystack” and other RAG/research benchmarks. The idea is that if the information exists in your Ally database, it will be found when needed.

Early progress is incredible, Ally already features one of the most sophisticated RAG implementations possible, and performance will only improve over time. This update includes:

• Parallel search tool for agents - In pursuit of the goal to shave every possible millisecond of latency off chat requests, we’ve parallelized search across multiple context providers. Basically, this means that rather than doing a BM25 search, a graph search, etc sequentially, they can happen concurrently in a single call that also includes search variations. These context sources are then combined with a Reciprocal Rank Fusion (RRF) algorithm. All of this means Ally can find what it needs faster and respond to you sooner with the answer.

• Intelligent chunking - Not everything should be chunked the same and you may want to be able to view a document at different resolutions depending on what you’re looking for and why. Ally is now set up to intelligently handle documenting chunking during processing and retrieval without increasing database storage size.

• Graph memory improvements - Added new entity types and optimized threshold and parameter values for more useful connections and faster graph traversal for quicker response times.

But, now we have Ally handling chat inference, embedding inference, graph queries, and more. All of these are very resource intensive straining your CPU, GPU, and memory resources!

How can we make sure Ally remains fast and responsive with all of this happening?

New: Adaptive Resource Scheduler

Nothing will get your fan spinning like a model that’s too big for your machine. But, a local LLM like Ally with agentic background processes, agentic RAG, graph memory, embedding pipeline, etc? It’s like juggling bowling balls. Ally needed a way to ensure that chat messages between Ally and the user aren’t delayed by lower priority jobs like entity extraction and document embedding generation so the Adaptive Resource Scheduler was built featuring:

• Fast lane for chat - To keep the chat experience as close to real-time as possible (an extremely important product priority/obsession), all your chat requests get the highest priority. Behind the scenes, there’s constant juggling of background work like entity extraction, embedding generation, agentic workflows, and indexing gracefully yields across provider types and task contexts.
• Always ready when you need it - Ally now keeps inference models loaded until the app is backgrounded. Models are purged from memory after 4 minutes of inactivity to free up system resources for other apps. This keeps Ally snappy and responsive when needed while freeing resources when not needed.
• Plays well with others - Beyond managing its own processes, Ally is also aware of the resource health of your system and queues low priority processes until resources CPU, GPU, and memory resources are free. It even monitors thermal state and backs off when temperatures are rising.

New: Agentic capabilities

Don’t worry, Ally will never shove AI features down your throat, and they can be completely disabled in Settings if you just want a snappy, simple Markdown editor. The goal of Ally’s AI features is to have them “not suck” which means, roughly applications of AI that:

• Add value (not just slapping AI onto things “because”)
• Work reliably (not just “some of the time” or “most of the time”)
• Include a record of changes with easy undo (observability and human in control)
• Hidden, but accessible, and can be turned off (AI that stays out of your way)

AI doesn’t have to suck, it can be incredibly useful, it just needs to be applied carefully and thoughtfully. That means using AI for use cases its well-suited for, designing and testing workflows to meet quality, performance, and reliability thresholds, and using models that are appropriate for the job.

So, this update adds support for agent chat, agentic RAG, and agentic workflows and processes. Obviously, this opens a lot of doors for new automation and interaction capabilities and we think you’ll be pleased with how they’re implemented.

Much more to come on that.

Feedback, as always, is not just appreciated, it is craved, listened to, and often implemented.