Have you ever been in a situation where you need to store a gigantic amount of data in your Salesforce org? Are you tired of using third-party storage systems and writing web service calls to get the massive amount of data? Having to crunch those large numbers, though a reflection of your own success, can be troublesome and performance degrading. Salesforce has come to the rescue again by introducing big objects-which are objects with massive storage capabilities on the Salesforce platform itself. They provide consistent performance over a data set of the order of billions and are accessible through a standard set of APIs to your org or to an external system.
What exactly are Big Objects?
Big Objects are merely custom objects but with massive data handling capabilities. They can store at least billions of data in them. They are similar to custom objects in some ways but very different in others. The scale of data is what lets Big objects stand apart in the crowd.
Just like normal Salesforce objects, Big Objects are also standard as well as custom. An example of a standard big object is FieldHistoryArchive, which stores archived field history data for Field Auditing purposes. Custom big objects are created and deployed by you using the Metadata API.
When can you use a Big Object?
As a massive data store, Big Objects can be a great help in many cases. Some typical uses of Big Objects are as follows:
- Audits and Tracking purposes
- Better customer experiences
- Historical Data Archive
Defining Big Objects
We can define big objects either using the Metadata API or in the Setup.
To create a new Big Object, go to Setup and search for Big Object in the quick find box. Then click on New and give the name for the big object.
All big objects are suffixed by “__b” at the end of the API Name. Lets consider we have a big object called Orders, the API name will Orders__b.
We can create custom fields of the following types in a big object.
- Lookup Relationship
- Date/ Time
- Number
- Phone
- Text
- Text Area (Long)
- URL
Before we can use the big object, we have to create an Index that will act as a composite primary key. This index will be used for querying and filtering the big object data.
Considerations for creating an Index for a big object
- Only the required custom fields can be included while creating an index.
- Up to a maximum of 5 fields can be included in the index.
- The total character length across all fields included in an Index cannot exceed 100.
- URL and Long Text Area fields cannot be included in the Index.
- Once we create an Index, we cannot modify or delete it. If we need to make any modifications we have to delete the big object and create it again.
- We have to specify the index position in the order in which we want to use the filters in the query. For example, we have to specify the Index position as 1 for the most commonly used filter in the query.
How to query a Big Object?
Obviously, keeping all that massive data in your org would be a waste if you can’t process it. Well, you can. To query the data stored in a big object, you can use a standard SOQL query or Async SOQL.
Using SOQL to query big objects has a disadvantage of large waiting times if the returned data set is large. Hence, it is advisable to use standard SOQL only if the query will return a small data set. For returning large sets of data, we must use Async SOQL. Now as is evident from the name, Async SOQL runs asynchronously. It is a way to run SOQL queries in situations where you can’t wait for the results in real-time due to the massive data that is being queried.
Async SOQL features:
- Highly scalable solution with the ability to process billions of data
- Ability to run multiple queries in the background while continuously monitoring their status
- Easy to use for someone coming from a SOQL background due to usage of a subset of SOQL commands
Below is how asynchronous SOQL looks like, we need to provide SOQL, and then target table and mapping between selected field and target table’s field.
Considerations While Querying Big Objects
- We cannot include the !=, LIKE, NOT IN, EXCLUDES, and INCLUDES operators in the query.
- Aggregate functions are not allowed.
- To retrieve a list of records, we must not include the Id field in the query. Including the Id field in the query will return results that have an empty Id.
- We can use =, <, >, <=, or >=, or IN operators on the last field in your query. Any prior fields in your query can use only the = operator.
Limitations
As limitless as they may seem; just like every other feature present in Salesforce, Big Objects have their limitations.
- You can create up to 100 big objects in your org. Although, this can vary depending on the license you have.
- Big objects are supported in Unlimited, Enterprise, Performance, and Developer editions up to 1 million records. Additional records capacity can be availed as an add-on license.
- Well, no UI or pointing & clicking here. You can only create big objects through the Metadata API.
- For the creation of records, you must use Apex or bulk DML operations. No UI is present for record addition.
- Currently Big Object only support fields and permission, We can not have
a. triggers
b. page layouts
c. extensive SOQL (indexed SOQL is supported but that’s extremely limited – and make sense as we are dealing with humongous data set)
d. no workflows, process builders, etc..
e. no reports
Conclusion
In general, developers need data for creating apps, but the volume of data is always a problem that they face. Big Objects help you deal with huge amount of records and run queries designed for consistent performance at scale. It enables the Big Data capability to the Salesforce platform. Big Objects helps you to manage data with the same Salesforce platform tools and APIs you know.