@whop-apps/feed

The Feed Component (tm)

High Level Architecture

The "feed component" consists of 3 parts:

• a backend api that stores posts, reactions and feeds
• a frontend client that fetches this data, manages the state of posts on the client
• a backend client that allows apps to run server logic before performing mutations

Backend API

The backend API is a rest api deployed on Cloudflare workers, that connects directly to dynamo db via https as its only data source. It is tightly integrated with the whop api v5 for authentication / user queries.

Post + Feed + Reaction structure

To interact with the feed component, you must perform requests on behalf of a "whop app". This can be done either "server to server" through the use of an api key, or through a user token. generated on behalf of an app.

• Each app owns many "feeds".
• Each feed has a collection of "posts"
• Each post, can optionally contains a collection of "children", which are posts, creating an infinite tree.
• Each post may have many reactions. Each reaction is of a certain type. (eg: reaction: "view" | "like" | "dislike")
• A user may only react to a post once per reaction type.

Models

There are currently 3 conceptual models stored in dynamo.

Feed Model:

A feed is the parent object for posts. It is owned and scoped to a single whop app. This means feeds created by different whops apps will never collide. (In general, all resources are scoped to app_id)

Each feed has 2 cached counters. Both relate to the amount of posts posted to the feed. The root_post_count tracks only posts which are direct children of the feed (ie: NOT nested posts aka comments) whereas the post_count field tracks the total count of all posts within the feed.

Each feed also a custom property field. This allows apps to attach custom configuration to each feed. For example this could include custom permission control setting purely for some feeds. The schema of this data can be specified using the schema_id field. This is to allow an app creating 2 different feeds, with different schemas for custom data.

Note: custom is treated as a fully opaque object, no validation or inspection is performed on this by the backend api.

The partition key of a Feed is it's owning app_id. The sort key is it's feed id. This allows us to potentially list all feed for an app efficiently. However, the usual access pattern is to get a single feed by passing both app id (derived from auth token) and the feed_id.

The feed_id is allowed to be fully custom - ie: you can make it an experience_id, a company_id or whatever else. This should allow a lot of flexibility.

Finally, each feed also has a read_access field. This is an access string. If the feed (or it's content's) are requested by a client that only provides a user level token, an access check is performed for that user against their user id.

Post Model:

The post model stored data about each post that is made within a feed.

It tracks basic data such as created_at, updated_at, user_id (whoever posted it), and custom + schema_id (similar to feed)

Queries and sorts.

There are 4 confusing properties. feed_sort, parent_id, root_feed_id and feed_id.

• parent_id is always set to the post_id of the parent post. If the post is a root level post (ie, direct descendant of a feed) the parent_id is undefined.
• root_feed_id is the actual id of the feed entity that the post is a child or grandchild of.

feed_id and feed_sort exist to support 2 different storage mechanisms for children posts. "default" and "inline".

To query a list of posts efficiently as a "feed" we use the GSI-1. The partition key here is the feed_id and the sort key is the feed_sort property.

In the default mode, these values are set in the following way:

• feed_id: is the id of the "list" that the post was posted in.
  • For top level posts this is set to the id of the actual Feed entity
  • For children posts this is set to child:{parent.post_id}
• feed_sort: is set to post_id. Since post ids are lexographically sortable by created at, querying children returns them in the order of recency.

However, in some feeds we would like to show a preview of a few children in the main list of posts. This is where "inline" mode comes in. In Inline mode, children posts are effectively "inlined" into the feed of the parent. This allows you to receive both posts, and children in a single. This can occur at any level of the comments tree. Posts can also be inlined multiple levels. Here:

• feed_id: is the exact same as the feed_id of the parent post. (this ensures that the post is return in the same dynamodb query).
• feed_sort: is set to the feed_sort of the parent post concatenated with :{post_id} of the current post. This ensure that the inlined post will be returned next to the parent one in a range query.

Essentially inlined posts should be used when you want the UI to look like this:

List: 
Main Post: post_01ax (feed_id=exp_xx, feed_sort=post_01ax)
- comment: post_01bx (feed_id=exp_xx, feed_sort=post_01ax:post_01bx) 
- comment: post_02cx (feed_id=exp_xx, feed_sort=post_01ax:post_02cx) 
- comment: post_03dx (feed_id=exp_xx, feed_sort=post_01ax:post_03dx) 

Main Post: post_02ex (feed_id=exp_xx, feed_sort=post_02ex) 
- comment: post_01fx (feed_id=exp_xx, feed_sort=post_02ex:post_01fx) 

Main Post: post_03gx (feed_id=exp_xx, feed_sort=post_03gx) 
- comment: post_01hx (feed_id=exp_xx, feed_sort=post_03gx:post_01hx) 
- comment: post_02ix (feed_id=exp_xx, feed_sort=post_03gx:post_02ix) 

But when your UI looks like this, default (aka: undefined) mode should be used :

List:
Main Post: post_01ax (feed_id=exp_xx, feed_sort=post_01ax)
<a>Open Post </a>

Main Post: post_02ex (feed_id=exp_xx, feed_sort=post_02ex) 
<a>Open Post </a>

Main Post: post_03gx (feed_id=exp_xx, feed_sort=post_03gx) 
<a>Open Post </a>

-------------------

Post Detail Page:
Main Post: post_01ax (feed_id=exp_xx, feed_sort=post_01ax)
- comment: post_01bx (feed_id=child:post_01ax, feed_sort=post_01bx) 
- comment: post_02cx (feed_id=child:post_01ax, feed_sort=post_02cx)
- comment: post_03dx (feed_id=child:post_01ax, feed_sort=post_03dx)

This concept is NOT exposed to the api clients and browser clients. In the outward facing api: feed_id is always the internal root_feed_id and feed_sort is not exposed. See the whop-feed/src/serializers/*.ts to see how objects are exposed.

Deletion

When a request is made to delete a post, the following checks happen.

• If the post has no children - immediately DELETE the row from the DB.
  • If: the current post has no siblings, AND the parent post is marked as deleted, delete the parent post, and cascade upward the tree.
• If the post has children - mark the post as deleted by setting deleted_at to the current UNIX timestamp in milliseconds.

Profile Info

When creating a post the user info is cached on the post record (user_name, profile_pic, display name etc...). This causes a bug when users change the profile details, resulting in their info in feed apps not updating.

Post Read Access

Similar to "feed" read access, there is also a field on Post for read_access. This field may either be a string or an object.

If it is undefined, read access is determined by the field of the posts' root_feed. Even if this field is present, users who do not have access to the feed cannot have access to the post.

If if is a string, then the entire post is invisible to a user who does not have access to the access string.

Otherwise, basic information about the post will be allowed. But there will be 2 keys within the object.

• content controls access to the content field of the post. If not provided, nothing is returned.

• custom: similarly to content this can be a string, and if access is not present, the custom data is not returned.

  • However, this can also be an object that specifies an access string per top level key of the custom data object.

This feature was build to allow for selling access to single posts.

Reaction Model:

A reaction is a record unique per user, per reaction type and per post. Ie: I cannot "like" a single post twice, however i can "like" the post and "heart" the post.

Hence a reaction is uniquely identified by it's post_id, user_id and reaction

Similarly to Post and Feed, reaction also has custom and schema_id which allows storing custom data on reactions.

Indexes

A reaction lives in 2 indexes.

• The primary one allows us to query a list of reactions by post id. effectively allowing us to see "who" reacted to a post.

• The secondary index allows us to query a list of reactions on a feed, partitioned by user. This is so that when fetching a "feed" of posts for a particular user, we can also directly fetch a list of their reactions for the posts contained within the feed. We need to do this so we can indicate to the user that they have already "liked" a post.

Counts

Reaction counts are cached on a ReactionCount record. This is so that frequent updates on dynamo do not need to write to a potentially large post record. Writing to smaller rows is cheaper.

When reacting to a post, a dynamo transaction is used to update the count and create the reaction in a single step. If the create fails, the count increase is aborted too.

Queries.

There are a handful of ways to query the data in the feed component from an api.

• /feeds/{feed_id} return information about a single feed
• /feeds/{feed_id}/posts returns posts, users (who made the posts) and my reactions for the posts (if signed in) within a feed.
  • if posts are inlined, it also returns those.
• /posts/{post_id} return information about a single post
• /posts/{post_id}/children same as the feeds/posts route, except its for children of the selected post.
• /posts/{post_id}/reactions returns users and reactions that were created for the current post.

This api is defined by an autogenerated OpenAPI v3 schema. We use hono as the api route handler, as this allows us to easily parse and validate body, search and path params as well as type the return type of the api.

When returning a response of potentially many items, the response is always formatted as a list of normalized entities, disambiguated by a _t field on each entity. This is to make it easy for the client to insert the response into the entity pool.

We can always add an option to format the list of items differently depending on a target client.

All queries ensure that the client has the correct permission to view items. This allows us to query data directly from the frontend. This is enabled by fields like read_access on Feed and Post.

Mutations

All mutations to the dynamo are submitted through the /actions endpoint as an array of Action objects. These actions are executed sequentially (since we don't know if an action depends on a previous one).

An action is executed by ActionFunction . This function receives the validated action body as well as the request context and will perform the necessary dynamo db updates to the underlying database models.

After updating the database base models, each action returns a list of "patches" or "updates" to the "Entity Pool" entities. These updates can either "put" or "destroy" and entity in the pool. This list of updates always pertains to a specific feed.

These the action results - aka updates are then stored in an Action model in dynamo db.

The partition key is the feed_id that the updates happened on (returned by the ActionFunction). The sort key is the action_id (another lexographically sortable id) The list of updates that happened during execution of the action function is also stored on the action row.

Furthermore a ttl field is stored, meaning this action is removed after 24hrs.

Clients may poll the actions endpoint to receive updates for the feed they are currently looking at.

Clients can then apply the changes directly to their local state.

The currently available mutations are

• patch_feed: create or update a feed (custom_data, schema_id, and read_access) can be updated.
• create_post: create a new post
• update-post: update a post - this sets updated_at and allows you to keep track of whether a post was updated already.
• delete-post: deletes a post (or marks it as deleted if it has children)
• set-reaction: creates or updates a reaction.
• remove-reaction: delete the reaction;.

Frontend Client

The frontend client is designed to hold all of the entities (feed, post, reaction) in memory in a normalized data structure, and make it easily readable from react.

Essentially this is the flow of data around the entire system:

• The frontend client holds a bunch of entities (feed, post, reaction, user) in a pool (entity pool).
• These entities are "subscribable", essentially allowing react components to subscribe to changes that happen on the entity.
• The pool is manipulated by "actions" which are objects that can be applied and reverted. Actions are submitted to the backend client.
• They can either be approved / denied here, then they are forwarded to the backend api, which applies the actions to the database.
• Actions mutate entities, and these entities are stored in an "action" table.
• The frontend client directly queries these actions, and applies the changes to its local version of the pool.

Entity

All entities inherit from BaseEntity. This base class provides shared functionality such as

• subscribers - the ability for react component to subscribe to changes for a particular object.
• local_data and server_data. This concept allows the client to "stage" some updates client side, and efficiently update this data locally. We can the submit an action which will "serverPatch" this client data onto the object.
• methods to easily read and write to both server and local data.
• references to the entity pool
• base methods to allow actual entities to validate and parse entities from raw json object.

Each actual entity then implements 4 types of things on top.

• Implement and override functions like primaryKey(), cursor() and parsing and validation functions.
• Actually implement getters and setters for all the properties that are readable and writeable on the entity.
• Implement helper functions that allow client to easily perform actions on the entity. (Performing actions = queueAction in the EntityPool's ActionQueue)
• Implement functions that Query other entities from the entity pool to simulate "relationships"

Query

Queries allow you to easily query a list of items from the entity pool.

• Queries DO NOT store items, they merely store references to items stored in the entity pool.
• Queries do not need to be able to fetch items remotely, but they can.
• Queries have a start and end cursor to be able to potentially paginate in both directions. WARN - this is not yet fully implemented by the api layer.
• Queries also have subscribers, this allows us to useFeedQuery and reactively rerender components when the list of items in a query changes.
• Queries are computed lazily - only when created do they start monitoring the pool
  • TODO: it should be safe to remove queries from the pool if they have 0 subscribers. (This means no one cares about the results)

A query is defined by a QueryDefinition:

export type QueryDefinition<E, T> = {
	// Which entity owns this instance of this query
	ownerId: string;
	
	// Allow you to pass a custom key / identifier / cache key for the query
	type: string;
	
	// This is a function, which given ANY entity in the pool,
	// returns true iff the entity should be included in the query's items.
	matcher: QueryMatcherDef<E>;
	
	// Standard JS sort function
	sorter: SortFunction<T>;
	
	// How can this query fetch initial or
	// more items that match the above matcher, and sorter. 
	fetcher?: ListFetchFunction;
	
	// Determines how we execute the fetcher when 
	// we use this query for the first time, 
	initialFetchMode?: "none" | "always" | "only-if-pool-empty";
	
	/* On Create, should we hydrate this query from the pool automatically using the provided matcher.  Default = true*/
	hydrateOnCreate?: boolean;
};

Queries should not be created by themselves - the EntityPool.query function takes care of creating a new query ONLY IF an existing one (cached via a combination of ownerId and type) has not already been created. This is to avoid refetching data too many times and to cache the result of queries.

Backend Client

The backend client is a very thin wrapper around the rest api allowing app developers to easily "accept" or "deny" actions, as well as perform side effects on success.

Custom types

To allow developers to use this component for more than just basic "text" posts, the component exposes the concept of "custom" data, stored as JSON inside the DynamoDB row. Custom data is able to be stored on post and reaction and feed.

A schema is able to be defined for each of these fields. This is done using createFeedDefinition. This definition is able to fully encode the entity relationships from above. : a feed of type A may contain posts of type B, but not posts of type C. Posts of type C are only able to be stored as "children" of posts of type B

Data submitted for these custom fields, is validated by the backend client. The frontend client is designed to be "typesafe" wrt to these types out of the box.