Source Functions
Source functions allow you to gather data from any third-party applications without worrying about setting up or maintaining any infrastructure.
All functions are scoped to your workspace, so members of other workspaces cannot view or use them.
Functions is available to all customer plan types with a free allotment of usage hours. Read more about Functions usage limits, or see your workspace’s Functions usage stats.
Create a source function
- From your workspace, go to the Catalog and click the Functions tab.
- Click New Function.
- Select Source Function and click Build.
Tip: Want to see some example functions? Check out the templates available in the Functions UI, or in the open-source Segment Functions Library. (Contributions welcome!)
When you click Build, a code editor appears. Use the editor to write the code for your function, configure settings, and test the function’s behavior.
Code the source function
Source functions must have an onRequest()
function defined.
This function is executed by Segment for each HTTP request sent to this function’s webhook.
async function onRequest(request, settings) {
// Process incoming data
}
The onRequest()
function receives two arguments:
request
- an object describing the incoming HTTP request.settings
- set of settings for this function.
We’ll learn more about settings later, let’s dive into how we can process the incoming request first.
To parse the JSON body of the request, use request.json()
method as in the example below.
async function onRequest(request) {
const body = request.json()
console.log('Hello', body.name)
}
Use the request.headers
object to get values of request headers.
Since it’s an instance of Headers
, the API is the same in both the browser and in Node.js.
async function onRequest(request) {
const contentType = request.headers.get('Content-Type')
const authorization = request.headers.get('Authorization')
}
To access the URL details, refer to request.url
object, which is an instance of URL
.
async function onRequest(request) {
// Access a query parameter (e.g. `?name=Jane`)
const name = request.url.searchParams.get('name')
}
Sending messages
You can send messages to the Segment API using the Segment
object:
async function onRequest(request) {
Segment.identify({
userId: 'user_id',
traits: {
name: 'Jane Hopper'
}
})
Segment.track({
event: 'Page Viewed',
userId: 'user_id',
properties: {
page_name: 'Summer Collection 2020'
}
})
Segment.group({
groupId: 'group_id',
traits: {
name: 'Clearbit'
}
})
Segment.set({
collection: 'products',
id: 'product_id',
properties: {
name: 'Nike Air Max'
}
})
}
Identify
Use Identify calls to connect user with their actions, and to record traits about them.
Segment.identify({
userId: 'user_id',
traits: {
name: 'Jane Hopper'
}
})
The Segment.identify()
method accepts an object with the following fields:
userId
- Unique identifier for the user in your database.anonymousId
- A pseudo-unique substitute for a User ID, for cases when you don’t have an absolutely unique identifier.traits
- Object with data about or related to the user, likename
oremail
.context
- Object with extra information that provides useful context, likelocale
orcountry
.
Track
Track calls record actions that users perform, along with any properties that describe the action.
Segment.track({
event: 'Page Viewed',
userId: 'user_id',
properties: {
page_name: 'Summer Collection 2020'
}
})
The Segment.track()
method accepts an object with the following fields:
userId
- Unique identifier for the user in your database.anonymousId
- A pseudo-unique substitute for a User ID, for cases when you don’t have an absolutely unique identifier.properties
- Object with data that is relevant to the action, likeproduct_name
orprice
.context
- Object with extra information that provides useful context, likelocale
orcountry
.
Group
Group calls associate user with a group - be it a company, organization, account, project, team or other.
Segment.group({
groupId: 'group_id',
traits: {
name: 'Clearbit'
}
})
The Segment.group()
method accepts an object with the following fields:
groupId
- Unique identifier for the group in your database.traits
- Object with data that is relevant to the group, likegroup_name
orteam_name
.context
- Object with extra information that provides useful context, likelocale
orcountry
.
Page
Page calls record whenever a user sees a page of your website, along with any other properties about the page.
Segment.page({
name: 'Shoe Catalog',
properties: {
url: 'https://myshoeshop.com/catalog'
}
})
The Segment.page()
method accepts an object with the following fields:
userId
- Unique identifier for the user in your database.anonymousId
- A pseudo-unique substitute for a User ID, for cases when you don’t have an absolutely unique identifier.name
- Name of the page.properties
- Object with information about the page, likepage_name
orpage_url
.context
- Object with extra information that provides useful context, likelocale
orcountry
.
Screen
Screen calls record when a user sees a screen, the mobile equivalent of Page, in your mobile app.
Segment.screen({
name: 'Shoe Feed',
properties: {
feed_items: 5
}
})
The Segment.screen()
method accepts an object with the following fields:
userId
- Unique identifier for the user in your database.anonymousId
- A pseudo-unique substitute for a User ID, for cases when you don’t have an absolutely unique identifier.name
- Name of the screen.properties
- Object with data about the screen, likescreen_name
.context
- Object with extra information that provides useful context, likelocale
orcountry
.
Alias
The Alias call merges two user identities, effectively connecting two sets of user data as one.
Segment.alias({
previousId: 'old-email@example.com',
userId: 'new-email@example.com'
})
The Segment.alias()
method accepts an object with the following fields:
previousId
- Previous unique identifier for the user.userId
- Unique identifier for the user in your database.anonymousId
- A pseudo-unique substitute for a User ID, for cases when you don’t have an absolutely unique identifier.
Set
The Set call uses the object API to save object data to your Redshift, BigQuery, Snowflake or other data warehouses supported by Segment.
Segment.set({
collection: 'products',
id: 'product_id',
properties: {
name: 'Nike Air Max 90',
size: 11
}
})
The Segment.set()
method accepts an object with the following fields:
collection
- Collection name.id
- Object’s unique identifier.properties
- Object with free-form data.
Runtime and dependencies
Functions use Node.js 14.x.
Functions do not currently support importing dependencies, but you can contact Segment Support to request that one be added.
The following dependencies are installed in the function environment by default.
atob v2.1.2
exposed asatob
aws-sdk v2.488.0
exposed asAWS
btoa v1.2.1
exposed asbtoa
form-data v2.4.0
exposed asFormData
@google-cloud/automl v2.2.0
exposed asgoogle.cloud.automl
@google-cloud/bigquery v5.3.0
exposed asgoogle.cloud.bigquery
@google-cloud/datastore v6.2.0
exposed asgoogle.cloud.datastore
@google-cloud/firestore v4.4.0
exposed asgoogle.cloud.firestore
@google-cloud/functions v1.1.0
exposed asgoogle.cloud.functions
@google-cloud/pubsub v2.6.0
exposed asgoogle.cloud.pubsub
@google-cloud/storage v5.3.0
exposed asgoogle.cloud.storage
jsonwebtoken v8.5.1
exposed asjsonwebtoken
lodash v4.17.15
exposed as_
moment v2.26.0
exposed asmoment
node-fetch v2.6.0
exposed asfetch
oauth v0.9.15
exposed asOAuth
stripe v8.115.0
exposed asstripe
xml v1.0.1
exposed asxml
xml2js v0.4.23
exposed asxml2js
Only the crypto
Node.js module is included (exposed as crypto
). Other built-in Node.js modules are not available.
Caching
Per-function global caching is available in the cache
namespace. The following functions are available:
cache.load(key: string, ttl: number, fn: async () => any): Promise<any>
- Obtains a cached value for the provided
key
, invoking the callback if the value is missing or has expired. Thettl
is the maximum duration in milliseconds the value can be cached. If omitted or set to-1
, the value will have no expiry. There is no guarantee that a value will be retained in the cache for the provided duration, however. The cache space is limited, so efforts to minimize the cached value size will afford a higher cache hit ratio.
- Obtains a cached value for the provided
cache.delete(key: string): void
- Forcefully remove the value associated withe the
key
.
- Forcefully remove the value associated withe the
The following example gets a JSON value through the cache, only invoking the callback as needed:
const ttl = 5 * 60 * 1000 // 5 minutes
const val = await cache.load("mycachekey", ttl, () => {
const res = await fetch("http://echo.jsontest.com/key/value/one/two")
const data = await res.json()
return data
})
Create settings and secrets
Settings allow you to pass configurable variables to your function, which is the best way to pass sensitive information such as security tokens. For example, you might use settings
as placeholders to use information such as an API endpoint and API key. This way, you can use the same code with different settings for different purposes. When you deploy a function in your workspace, you are prompted to fill out these settings to configure the function.
First, add a setting in Settings tab in the code editor:
Click Add Setting to add your new setting.
You can configure the details about this setting, which change how it’s displayed to anyone using your function:
- Label - Name of the setting, which users see when configuring the function.
- Name - Auto-generated name of the setting to use in function’s source code.
- Type - Type of the setting’s value.
- Description - Optional description, which appears below the setting name.
- Required - Enable this to ensure that the setting cannot be saved without a value.
- Encrypted - Enable to encrypt the value of this setting. Use this setting for sensitive data, like API keys.
As you change the values, a preview to the right updates to show how your setting will look and work.
Click Add Setting to save the new setting.
Once you save a setting, it appears in the Settings tab for the function. You can edit or delete settings from this tab.
Next, fill out this setting’s value in Test tab, so that we can run our function and verify correct setting value is being passed.
Note, this value is only for testing your function.
Now that we have our setting set up and test value filled in, we can add code to read its value and run our function:
async function onRequest(request, settings) {
const apiKey = settings.apiKey
//=> "super_secret_string"
}
When you deploy a source function in your workspace, you are prompted to fill out settings to configure the source. You can access these settings later by navigating to the Source Settings page for the source function.
Test the source function
You can test your code directly from the editor in two ways: either by receiving real HTTP requests through a webhook, or by manually constructing an HTTP request from within the editor.
The advantage of testing your source function with webhooks is that all incoming data is real, so you can test behavior while closely mimicking the production conditions.
Testing source functions with a webhook
You can use webhooks to test the source function either by sending requests manually (using any HTTP client such as cURL or Insomnia), or by pasting the webhook into an external server that supports webhooks (such as Slack).
From the source function editor, copy the webhook URL from the “Auto-fill via Webhook” dialog. To trigger the source function, send the request using the POST
method, with the Content-Type
header set to application/json
or application/x-www-form-urlencoded
.
Testing source functions manually
You can also manually construct headers and body of an HTTP request right inside the editor and test with this data, without using webhooks.
Save and deploy the function
Once you finish building your source function, click Configure to name it, then click Create Function to save it.
Once you do that, the source function appears on the Functions page in your workspace’s catalog.
If you’re editing an existing function, you can Save changes without updating instances of the function that are already deployed and running.
You can also choose to Save & Deploy to save the changes, and then choose which already-deployed functions to update with your changes. You might need additional permissions to update existing functions.
Source functions logs and errors
Your function might encounter errors that you missed during testing, or you might intentionally throw errors in your code (for example if the incoming request is missing required fields).
If your function throws an error, execution halts immediately. Segment captures the incoming request, any console logs the function printed, and the error, and displays the this information in the function’s Errors tab. You can use this tab to find and fix unexpected errors.
Functions can throw an Error or custom Error, and you can also add additional helpful context in logs using the console
API.
For example:
async function onRequest(request, settings) {
const body = request.json()
const userId = body.userId
console.log('User ID is', userId)
if (typeof userId !== 'string' || userId.length < 8) {
throw new Error('User ID is invalid')
}
console.log('User ID is valid')
}
Warning: Do not log sensitive data, such as personally-identifying information (PII), authentication tokens, or other secrets. You should especially avoid logging entire request/response payloads. Segment only retains the 100 most recent errors and logs for up to 30 days but the Errors tab may be visible to other workspace members if they have the necessary permissions.
Error types
- Bad Request is any error thrown by your code not covered by the other errors.
- Invalid Settings: A configuration error prevented Segment from executing your code. If this error persists for more than an hour, contact Segment Support.
- Message Rejected: Your code threw
InvalidEventPayload
orValidationError
due to invalid input. - Unsupported Event Type: Your code does not implement a specific event type (
onTrack()
, etc.) or threw aEventNotSupported
error. - Retry - Your code threw
RetryError
indicating that the function should be retried.
Segment only attempts to run your source function again if a Retry error occurs.
Managing source functions
Source functions permissions
Functions have specific roles which can be used for access management in your Segment workspace.
Access to functions is controlled by two permissions roles:
- Functions Admin: Create, edit and delete all functions, or a subset of specified functions.
- Functions Read-only: View all functions, or a subset of specified functions.
You also need additional Source Admin permissions to enable source functions, connect destination functions to a source, or to deploy changes to existing functions.
Editing and deleting source functions
If you are a Workspace Owner or Functions Admin, you can manage your source function from the Functions tab in the catalog.
Connecting source functions
You must be a Workspace Owner or Source Admin to connect an instance of your function in your workspace.
From the Functions tab, click Connect Source and follow the prompts to set it up in your workspace.
Once configured, find the webhook URL - either on the Overview or Settings → Endpoint page.
Copy and paste this URL into the upstream tool or service to send data to this source.
Source function FAQs
What is the retry policy for a webhook payload?
The webhook payload retries up to 5 times with an exponential backoff for the function in the event of a failure with the function. After 5 attempts, the message is dropped.
What is the maximum payload size for the incoming webhook?
The maximum payload size for an incoming webhook payload is 512 KiB.
What is the timeout for a function to execute?
The execution time limit is 5 seconds, however Segment strongly recommends that you keep execution time as low as possible. If you are making multiple external requests you can use async / await to make them concurrently, which will help keep your execution time low.
This page was last modified: 18 May 2021
Need support?
Questions? Problems? Need more info? Contact us, and we can help!