How can I create custom CDR rules with advanced DSL search queries?

Now that Agari Phishing Defense has added local rules to its Continuous Detection and Response (CDR) functionality, you can create your own fine-grained local rules by creating custom search queries with the available Domain Specific Language (DSL). These search queries are to find and match messaged with the characteristics you define with the DSL. This article defines the components of the DSL available in Agari Phishing Defense and provides some examples of queries using the DSL.

You enter search queries in the Rule Description field on the Local CDR Rule page in Agari Phishing Defense. See Agari Phishing Defense Administrator & User Guide for more information about CDR rules.

Syntax

A search query always contains one or more search expressions of field operator operand, with each additional search expression delineated by either an AND or an OR conjunction. Specifically, the syntax is:

field operator operand {AND/OR field operator operand...}

The default order of precedence when evaluating multiple search expressions in a query is left-to-right. You can use group search expressions in parentheses to control precedence of evaluation. Wildcards are not allowed. You should use quotes around all string operands for clarity and consistency. (Technically they are not required for operands that do not contain spaces or reserved characters, such as the operator characters = and >, but to avoid the possibility of errors, always use them to enclose string operands.) You must not use quotes around a number or date operand.

Fields

A field represents a message characteristic that you can search for. You can use the following fields as the field component of a search expression. The Type indicates which operands you can use. For example, a field of type float can be used with any operand that can evaluate numbers.

Field	Type	Description
attachment_extension	string	An exact match of an attachment extension as a string. For example, doc will not match docx. Do not include the "." (period, or dot) that typically precedes extensions in filenames.
attachment_filename	string	An exact match of an attachment filename.
attachment_sha256	string	An exact match of an attachment SHA256 hash.
attack_types	string	An attack type classification as either a string or a list. The following table shows the values you can use in a search expression with the attack_types field and the associated attack type: Search Expression Field Attack Type low_trust Spam or Graymail lookalike_domain Look-alike Domain spoof Domain Spoof dni Brand Display Name Imposter address_group Individual Display Name Imposter compromised_webmail Compromised Account likely_malicious_uri Likely Malicious URI malicious_attachments Likely Malicious Attachment covid_19 COVID-19
authenticity	float	Authenticity score, represented as a floating point value between 0.0 and 10.0.
spf_result	string	List of possible message SPF results. The following table shows the values you can use in a search expression with the spf_result field and the associated meaning for each (note that the values are all single characters): Search Expression Value SPF Result + SPF pass - SPF fail ~ SPF soft fail n No SPF record p SPF Permanent Error
dkim_result	string	List of possible message DKIM results. The following table shows the values you can use in a search expression with the dkim_result field and the associated meaning for each (note that the values are all single characters): Search Expression Value DKIM Result + DKIM pass n Not signed
dmarc_result	string	List of possible message DMARC results. The following table shows the values you can use in a search expression with the dmarc_result field and the associated meaning for each (note that the values are all single characters): Search Expression Value DMARC Result + DMARC pass - DMARC fail n Not available
domain_reputation	float	Reputation of sender domain, represented as a floating point value between 0.0 and 10.0.
domain_tags	string	An exact match of a domain tag, as either a string or a list.
from	string	The full From: in a message header.
from_domain	string	The domain portion within the full From: message header only.
from_to_match	boolean	Whether or not the From email address portion of the full header matches the To email address.
has_attachment	boolean	Whether or not message has an attachment.
has_attachment_malicious	boolean	Whether or not a message has an attachment that has been determined to be malicious.
ip	string	IP address or CIDR (see Classless Inter-Domain Routing for more information about CIDRs).
mail_from_domain	string	Mail from domain. The domain of the envelope sender. Matches only the entire mail from domain. (No partial matches.)
message_id	string	Globally unique message identifier. Matches only the entire message identifier. (No partial matches.)
message_trust_score	float	Message trust score, represented as a floating point value between 0.0 and 10.0.
reply_to	string	The full Reply-To in a message header.
reply_to_no_match	boolean	Whether the Reply-To address does not match the From address. (Use true in the operand to find when they do not match, and use false in the operand to find when they do match.)
sbrs	float	SenderBase Reputation Score, represented as a floating point value between -10.0 and 10.0.
start_date	date	A date in ISO 8601 format that defines the earliest time and date to match a message (not applicable for matching incoming messages). Example: To limit your search to anything after July 14th 2020 at 14:44 and 25 seconds UTC, add the following condition: start_date > 2020-07-14T14:44:25Z This field works only with the operators =, >, and >= Cannot be the only expression in a rule.
subject	string	Message subject. For example, if you enter a search expression of subject ANY "login" it would match the subject of "Change your login password." If you enter a search expression of subject eq "login"it would be an exact match on the subject of "login".
to	string	An exact match of the full To: in a message header.
uri	string	A URL anywhere in a message. Exact matches only.

Operators and Operands

An Operator defines what you can look for in a search expression and how you can look for it. Each operator in a search expression has specific operand types that are legal. Operators are divided into 2 types: numeric and string. Numeric operators are used for numeric comparisons. String operators are for finding and matching text.

The following tables list the available operators you can use in search expressions.

Numeric Operators

Operator	Description
=	field is equal to all or part of the operand value
!=	field is not equal to all or part of the operand value
>=	field is greater than or equal to the operand value
>	field is greater than the operand value
<	field is less than the operand value
<=	field is less than or equal to the operand value

String Operators

Operator	Description
	List Operators
in	field is in the operand list, where the operand list is one or more comma-separated values, enclosed in brackets
not in	field is not in the operand list, where the operand list is one or more comma-separated values, enclosed in brackets
	String Operators
any	any part of the field matches the entire operand
not any	no part of the field matches the entire operand

An Operand defines the value(s) you're looking in the field.

Types

Types refer to data types of the values that you can use in search expressions operands.

Type	Description
string	One or more characters. In search expressions, strings must be enclosed in quote marks. Strings that contains numeric values are not the same as numeric values.
float	A number that includes a decimal point.
integer	A number that does not include a decimal point.
date	A number in ISO 8601 date format. Note that this format contains non-mumeric characters, such as hyphens, but is still interpreted as a numeric type.
boolean	A binary value that is represented by true or false. Note that these are not considered strings and do not get enclosed in quote marks.
list	A complex type that allows you to combine multiple strings in a search expression operand using the following format: [”item 1”, ”item 2”].

Conjunctions

Conjunctions connect simple search expressions to create more complex search expressions.

Conjunction	Description
AND, &	Both must be true
OR, |	Either may be true

 

Escape Characters

Special characters such as `? [ & %` may need to be escaped as they represent other things in the code base, similar to most regex expressions.  

Wildcards such as `?` and `*` will need to be escaped by `^`

Example: If you have a URI with a `?` in the string, you will need to put a carrot `^` before it. 

Example: To include brackets as an exact match such as `[SPAM]` in the subject, you must do `\[Spam\]`

Errors

Phishing Defense will check for syntax errors in the search expressions before it tries to process them. There are also other errors that could occur. This section lists possible errors and describes what to look for in your search expression to resolve the error.

Phishing Defense will check for syntax errors in the search expressions before it tries to process them. There are also other errors that could occur. This section lists possible errors and describes what to look for in your search expression to resolve the error.

There was an error in your DSL syntax. Premature end of input near [snippet].

This means something was not closed, such a parenthesis or quote marks. Any grouping marks must be used in pairs, and this error signifies that the second instance of a grouping pair was not found. [snippet] is a section of the search expression as close as possible to the error.

There was an error in your DSL syntax. Unexpected input near [snippet].

This means an incorrect operand, a mismatched operand, or an extra closing parenthesis. "Incorrect" could mean a mis-typed operand, so check carefully for spelling errors using the information in this topic. [snippet] is a section of the search expression as close as possible to the error. A mismatched operand would be something such as to >= plorence@agari.com because the to field takes a string value, which cannot be evaluated as more than or equal to.

There was an error in your DSL syntax. Don't know what to do with [snippet].

This is the generic error message if an error is found and cannot be definitively determined to be one of the above errors. One reason this could occur is if you do not use the field operator operand pattern in your search expression. This could also occur for mis-typed fields or operators.

Wildcards not allowed

This means a wildcard character (*) was found in the operand. Wildcards are not allowed in search expressions.

OR statements not allowed in parentheses. Please rewrite your query.

Parentheses cannot enclose two or more search expressions that contain an OR conjunction.

For example, if you construct a query like this:

from_domain = foo.com and (subject = "x" OR subject = "y")

...you would need to reconstruct your query to be:

(from_domain = foo.com and subject= "x") OR (from_domain = foo.com and subject = "y")

 

Custom CDR rules cannot be created for results exceeding 2000 existing messages. Please refine your search criteria to return a smaller set of messages.

The search expression returned more than 2000 results. Add additional field operator operand constructions to limit the total number of search results. See the examples below.

[field name] is not a supported field.

This means that a field was entered as part of a search expression that is not one supported for CDR rules. Only the fields listed above can be used in CDR search expressions.

Lists are not a supported value for field: [field name].

In the search expression that uses the field, lists of values are not allowed as an operand. The search expression can have only a single value.

As an alternative, you can use multiple search expressions connected with an OR conjunction. So instead of this:

field = ["value1", "value2", "value3"]

You can rewrite the above like this:

field = "value1" OR field = "value2" OR field = "value3"

The value of [field name] must be numeric.

Only a numeric value is valid as an operand in search expressions with this field. Make sure that the operand is a number and that it is not enclosed in quotes. Values enclosed in quotes are considered strings.

The value of [field name] must be a string or valid list.

Only a string or a list of strings is valid as an operand in search expressions with this field. Make sure that any search expression with this field does not contain a numeric or boolean value and that the operand is enclosed in quotes.

The value of [field name] must be true or false.

Only a boolean value is valid as an operand in search expressions with this field. Make sure than any search expression with this field contains an operand value of either true or false only, and that the value is not enclosed in quotes.

The value of [field name] cannot be an empty string.

An empty string, or "", is not allowed in CDR rule search expressions.

search_dsl must contain at least one valid field for ongoing monitoring.

This is an error that something unexpected occurred after initially validating and saving the CDR rule. Contact Agari support for assistance.

Examples

This section provides examples of search expressions.

Internal Domains

You want to identify domains that are internal, that is, domains that you own, have control over, and trust. These are domains that you would have given the internal tag to. (Note that this is a search expression that can also be accomplished in message search. Anything that can be done with message search can also be done with DSL search expressions, although the latter can also do much more. This example demonstrates a very simple search expression.)

domain_tags = "internal"

This search expression will match any message that contains a domain that has been given the internal tag.

 

IP Address and Evaluation Results

You want to identify messages that include specific IP addresses and that also have an SBRS result.

(ip = "1.2.3.4" AND sbrs is not null) OR (ip = "2.3.4.5" AND sbrs is not null)

This search expressions will match any messages that contain either the IP address 1.2.3.4 or the IP address 2.3.4.5 and also contain an SBRS value.

User Receiving Attachments

You're concerned about attachments in messages sent to a specific user.

to = "jsmith@mycompany.com" AND has_attachment = "true"

This search expression will match any message sent to jsmith@mycompany.com that has an attachment of any kind.

You may also want to find only specific types of attachments.

to = "jsmith@mycompany.com" & has_attachment = "true" & attachment_extensions = "doc" OR attachment_extensions = "docx" OR attachment_extensions = "rtf"

This search expression will match any message sent to jsmith@mycompany.com that has an attachment and that attachment has an extension of doc, docx, or rtf, which are typical extensions of Microsoft Word documents.

Specific Terms With Low Reputation

You're concerned with messages you see that seem to be part of a specific campaign that you want to specifically block.

subject = "urgent" OR subject = "important" and domain_reputation <= 2 AND domain_tags != "service" OR domain_tags != "customer"  OR domains_tags ~= "social"

This search expression will match any messages with "urgent" or "important" in the subject line (note that this is not case-sensitive) that also has a low domain reputation and does not come from a domain that you have tagged as one of your trusted types.