RFC-0001 - Llama Stack

[raghotham]

As part of the Llama 3.1 release, Meta is releasing an RFC for ‘Llama Stack’, a comprehensive set of interfaces / API for ML developers building on top of Llama foundation models. We are looking for feedback on where the API can be improved, any corner cases we may have missed and your general thoughts on how useful this will be.

Ultimately, our hope is to create a standard for working with Llama models in order to simplify the developer experience and foster innovation across the Llama ecosystem.

[trholding]

General Opinion: I feel there is a need for a 3b or 4b base model that makes it easier / faster to do inference on very low end hardware or CPU (which is my goal). The benefits would be less space and resources needed at the cost of accuracy / quality, but fine tuning that for the end user's use case could be a game changer. If quantized the models would be smaller and may be less usable for text generation, however it still could be very good for finding probabilities for tokens and be used in applications such as compression of text (llm-zip), logs, binaries and structured data yielding high compression ratio.

[raghotham]

Thanks for your comment @trholding. Completely understand the need for smaller models. In this release, we have provided FP8 quantized versions for our largest model (405B). In addition, we have released both dynamic-fp8 inference code as well as inference code for fp8 persisted weights part of the inference implementation. Please check them out.

[mberman84]

Not sure this is the appropriate place for this, but a general question about the agentic framework: is it supposed to be a standalone agent framework or a language definition for other frameworks to plug into?

[TheUniversalAxiom]

The Universal Axiom Organic Transformer Model offers a transformative approach to enhance the Llama Stack API:

Dynamic Adaptation: Implement auto-scaling and continuous learning interfaces.
Balanced Growth: Incorporate Fibonacci-inspired scaling for system stability.
Ethical Alignment: Integrate ethical checkpoints and transparency layers.
Multidimensional Processing: Develop multi-scale attention mechanisms for complex temporal data.
Bias Mitigation: Implement an Axiomatic Subjectivity Scale for quantifying and reducing biases.
Holistic Integration: Design interconnected architectures reflecting Impulses, Elements, and Pressures.
Quantum-Inspired Algorithms: Enhance processing with probabilistic and entanglement-inspired methods.

This Axiom-based approach transforms Llama Stack into an evolving, ethically-aligned ecosystem that adapts to user needs and technological advancements.

For a deeper dive into how The Universal Axiom can revolutionize AI development and foster true innovation, visit:
https://www.epiphanyengine.ai

Discover how this framework can elevate your projects beyond traditional AI paradigms.

[tanliboy]

I would like to share my wish list for your considerations:

Models

• Restricted Prediction: It would be awesome to have native and efficient support for restricted prediction along with a pre-defined schema, such as JSON schema support in llama.cpp grammars. This could boost the usability of the models for generating reliable structured data.

Fine-tuning

• Continued Pretraining: Maintaining the original data distribution during continued pre-training is tricky. A usual approach is to mix sampled pre-training data with new training data. If Llama models could provide access to a sampled pre-training dataset, it would make this process a lot smoother and ensure consistency.

• Knowledge Distillation: The 405B model is amazing. I wish that we could have an E2E knowledge distillation tool/pipeline/API to fine-tune using token distributions from teacher models.

• RS/DPO: It is good to know that Llama3.1 has switched from RLHF to Rejection Sampling and Direct Preference Optimization (DPO) for optimizing preferences. It would be amazing to offer sample fine-tuning recipes or best practices that could help us fine-tune Llama3.1 models effectively.

Agentic

• MCTS + LLM: There have been some cool attempts to use Monte Carlo Tree Search (MCTS) to boost the logical reasoning capabilities of LLMs. However, there’s still a gap when it comes to tools or viable paths for tightly integrating MCTS with LLMs, except for some rumored projects in closed-source models. Creating a robust tool or interface for this integration would be a huge win for AI agent developers.

[anoopkatti]

OpenAI recently released the Assistant API.

1. An Assistant is an LLM equipped with some tools. This allows the assistant to automatically select the right set/sequence of tools given a user request. This means an AI developer doesn't need to define when to select what tools.
2. Moreover, a user chats with an assistant within a thread. This allows the Assistant to use the prior context (by default) when continuing to chat. In the absence of threads, an AI developer should include all the relevant prior context in the follow-up prompts - thus wasting tokens.

AFAIK, LLama already supports automatic tool usage. But, it would be great if it can also natively support threads.

[metaskills]

@anoopkatti Thanks for calling out the Assistants API. I've been doing research on it using a mini framework called Experts.js. As I understand it, any model that supports function calling should be able to support a multi ai agent system such as this. So about Threads and how that API works:

First, the API is very "cloud resource" oriented. Meaning, instantiating an assistant creates a resource which API uses on the backend during inference. Second, threads ( managed memory ) is a cloud resource. It is really cool too and I'm seeing it pop up in several places, like Amazon Bedrock a few days ago. I think this is important because knowing the boundaries might help Llama Stack know where to implement things. For example, were you thinking llama inference start would be where the managed thread happens?

[wamoyo]

Ideally the API matches OpenAI's and Anthropic's for interoperability reasons. Of course, adding additional things is great! : ) But, if at least the basic parts of the API can match the other frontier models, we can use them interchangeably collaboratively etc.

[bionicles]

I disagree with the need for the API to match OpenAI/Anthropic, because they're using arrays of structs instead of structs of arrays, and it's well established that the latter data structure is better for performance.

Status Quo: Array of Structs (Object Oriented)

dialogue = [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "What's the weather in SF?" },
]

Improved Paradigm: Struct of Arrays (Data Oriented Design)

dialogue = {
    "role": ["system", "user"],
    "content": ["You are a helpful assistant.", "What's the weather in SF?"]
}

in my experience, the performance benefit of this seemingly minor change can be vast, as there are way fewer memory allocations in the hot loop. If the number of messages is known in advance, the entire lists can be allocated upfront, for even greater benefit.

Yes, it is possible to pass the wrong length of list, but this can return a clear and helpful error message, and it's undoubtedly possible to screw up the other way, too.

[wamoyo]

Is there a good reason not to make both approaches available? At some level the problem of interoperability needs solving. I guess you see it better solved one level up in libraries?

Objects of arrays seems less human readable and harder to edit as the objects and arrays get bigger.

I disagree with the need for the API to match OpenAI/Anthropic, because they're using arrays of structs instead of structs of arrays, and it's well established that the latter data structure is better for performance.

Status Quo: Array of Structs (Object Oriented)

dialogue = [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "What's the weather in SF?" },
]

Improved Paradigm: Struct of Arrays (Data Oriented Design)

dialogue = {
    "role": ["system", "user"],
    "content": ["You are a helpful assistant.", "What's the weather in SF?"]
}

in my experience, the performance benefit of this seemingly minor change can be vast, as there are way fewer memory allocations in the hot loop. If the number of messages is known in advance, the entire lists can be allocated upfront, for even greater benefit.

Yes, it is possible to pass the wrong length of list, but this can return a clear and helpful error message, and it's undoubtedly possible to screw up the other way, too.

[geoffreya]

BEFORE installing the toolchain, it would be nice to be able to 1-Read the api docs and 2-Read the example notebooks that use the api. The project's readme in github would be a nice place to provide the link to those.

[bionicles]

Is there a good reason not to make both approaches available? At some level the problem of interoperability needs solving. I guess you see it better solved one level up in libraries?

perfectly reasonable to support both, and pandas/arrow/polars are up to the task