AI Glossary / Data engineering and architecture / Beginner
Object storage
Definition
Object storage is a way to store data as individually addressable objects, each containing data and associated metadata. Applications retrieve an object using its identifier or key within a container such as a bucket. It is widely used for images, documents, backups, analytical datasets and AI model files, especially for large collections managed through APIs.
What it means
An object is a stored unit, not an object in the object-oriented programming sense. A photograph, a compressed data partition or a model checkpoint can each be stored as one object. Metadata describes properties such as content type or application-defined information.
Object storage differs from block storage, which presents addressable blocks used by systems such as filesystems, and from a conventional shared filesystem interface. Some tools make object storage look like a folder tree or mounted drive. That convenience does not mean every filesystem operation has identical behavior or performance.
How it works
An application uploads data under a key, obtains it through a read operation and manages it using the service's supported interfaces. A container groups objects and provides a scope for configuration and access controls. Versioning, retention and lifecycle features may be available, but they require the appropriate service capabilities and configuration.
In a flat namespace, a key such as images/approved/photo7.jpg can contain slashes without representing ordinary nested filesystem directories. Some object-storage products also offer hierarchical namespace features, so it is better to check the specific storage type than claim all object storage has one universal layout.
Applications must understand update, listing, consistency and access behavior. For example, Amazon S3 documents strong read-after-write consistency for its object PUT and DELETE operations. Such a product guarantee should not be replaced with the outdated blanket claim that object storage is always eventually consistent.
Three practical examples
1. Website media: A publishing platform stores approved photographs as objects and keeps their captions, ownership and publication status in an application database. The storage key connects each database record to its media.
2. AI training: A team stores partitioned training files and records their exact identifiers in a dataset manifest. The manifest helps reproduce which inputs a training run actually used.
3. Backup archive: A business stores backup packages in an appropriate storage class and tests restoration. Retention settings and recovery procedures matter alongside the existence of the uploaded objects.
Think of it this way
Think of a warehouse of sealed packages. Each package has an addressable label and descriptive information, while an inventory helps people find the package they need. The warehouse can store many kinds of contents without understanding each one's business meaning.
The analogy has limits. Object systems provide digital APIs and may support features such as versioning or partial reads. A label that resembles a folder path does not necessarily represent a physical hierarchy, and storing a package does not automatically validate its contents or maintain an application-level catalog.
Worked example
Suppose a research project stores 1,000 objects averaging 2 megabytes each. Using decimal units for this illustration, the payload totals 2,000 megabytes, or 2 gigabytes. A manifest lists the 1,000 keys, their dataset split and the expected content identifiers.
The team replaces 100 images with new 2-megabyte versions while retaining the old versions. The stored image payload becomes 2,200 megabytes, excluding metadata and service overhead. Counting only the latest 1,000 images would understate retained storage.
For reproducibility, a training run should reference the intended versions or immutable names where the system supports them. A key that points to changing content is insufficient to establish the exact bytes used yesterday. Storage cost also depends on requests, retrieval, transfer and the selected service; this arithmetic estimates payload volume only.
Illustrative case study
Fictional case study: ObjectMeadow moves product photographs into object storage. Its first catalog contains descriptive filenames, but several teams upload replacements under the same keys. A historical campaign later displays updated photographs instead of the approved images.
The team introduces a publication manifest tied to immutable object names or explicit versions and records approval information separately. It checks whether each listed object is readable by the intended application and verifies content integrity during the publication workflow.
ObjectMeadow also tests a restore because storage availability alone does not prove recovery from an accidental application-level change. The project learns to separate durable storage, controlled publishing and reproducible references. Each solves a different part of the content-management problem.
Common misconceptions and mistakes
A common mistake is assuming that folder-like names guarantee filesystem semantics. Another is treating the storage service as a complete searchable catalog of business meaning. An object key can locate bytes without explaining their quality, provenance or permitted use.
Do not assume that durability prevents every deletion or overwrite, or that versioning is automatically enabled. Avoid presenting public access as a requirement for application access. Appropriate permissions can allow authorized services to use private objects. Finally, verify product-specific consistency and update rules rather than generalizing from an old description of a different service.
Business use
Object storage can support scalable media libraries, data lakes and AI workflows, but the useful design includes naming, metadata, access control and lifecycle decisions. A collection of files without an owner or manifest can become difficult to govern even when the storage system works correctly.
Teams should model access frequency, retrieval requirements, object sizes and recovery needs. Test representative reads and writes through the intended application path, including permission errors and missing objects. For AI projects, retain the relationship between stored data, dataset versions and model runs so that a result can be investigated and reproduced later.
People also ask
Can object storage contain structured data?
Yes. Structured formats such as analytical data files can be stored as objects alongside images and other content.
Are slashes in a key always real folders?
No. In a flat namespace they can simply be characters interpreted by tools as a convenient hierarchy.
Does storing an object make it public?
No. Access depends on the service's permissions and configuration; public publication is a separate choice.
Check your understanding
A model run records only a key whose contents are later overwritten. Does that key alone establish the original training input?
Answer: No. The workflow needs an immutable reference, explicit version or other reliable record of the exact content used.
Related terms
Learn next
Design a small dataset manifest containing object references, content identifiers and intended roles. Test whether you can retrieve the same inputs after later uploads and configuration changes.
More in Data engineering and architecture
- OAuth - OAuth is an authorization framework through which a client can obtain limited access to a protected service.
- Object Data Model - An object data model represents information through objects with identity, state and, in object-oriented approaches, associated behavior.
- Object - role modeling - Object - role modeling, often abbreviated ORM, is a conceptual data-modeling approach that describes a business domain through facts and the roles that...
- Observability (software) - Observability in software is the ability to understand a system's internal behavior by examining the information it exposes.
- OLAP cube - An OLAP cube is a multidimensional analytical model that organizes measures, such as sales or quantity, by dimensions such as product, location and time.
- One-to-many (data model) - A one-to-many relationship connects one instance of an entity type to potentially several instances of another, while each instance on the many side relates...
Sources
AWS, “What is Amazon S3?”: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
AWS, naming objects: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html
Google Cloud, “About Cloud Storage objects”: https://cloud.google.com/storage/docs/objects
Product documentation supports the object model and scoped examples of service behavior. Storage calculations and ObjectMeadow are fictional; no current price or universal performance guarantee is implied.
Previous: Object - role modeling Next: Objective