If we want to add new features in the kernel, it becomes difficult to preserve backward compatibility (binary and API), and at the same time to allow user-level processes to take advantage of the new features (if the kernel supports them).
One of the mechanisms that allows us to preserve the backward compatibility is a sort of negotiation between the user-level process and the kernel:
1.
The user-level process tries to enable in the kernel the set of new features (and the corresponding API) it would like to use.
2.
The kernel returns the (sub)set of features it knows about and is willing to be enabled.
3.
The user-level process uses only that set of features the kernel has agreed on.
To support backward compatibility, if the user-level process does not ask for any new features, the kernel defaults to the basic multicast API (see the
Programming Guide section). Currently, the advanced multicast API exists only for IPv4; in the future there will be IPv6 support as well.
Below is a summary of the expandable API solution. Note that all new options and structures are defined in
<netinet/ip_mroute.h> and
<netinet6/ip6_mroute.h>, unless stated otherwise.
The user-level process uses new
getsockopt()/
setsockopt() options to perform the API features negotiation with the kernel. This negotiation must be performed right after the multicast routing socket is open. The set of desired/allowed features is stored in a bitset (currently, in
uint32_t; i.e., maximum of 32 new features). The new
getsockopt()/
setsockopt() options are
MRT_API_SUPPORT and
MRT_API_CONFIG. Example:
uint32_t v;
getsockopt(sock, IPPROTO_IP, MRT_API_SUPPORT, (void *)&v, sizeof(v));
would set in
v the pre-defined bits that the kernel API supports. The eight least significant bits in
uint32_t are same as the eight possible flags
MRT_MFC_FLAGS_* that can be used in
mfcc_flags as part of the new definition of
struct mfcctl (see below about those flags), which leaves 24 flags for other new features. The value returned by
getsockopt(
MRT_API_SUPPORT) is read-only; in other words,
setsockopt(
MRT_API_SUPPORT) would fail.
To modify the API, and to set some specific feature in the kernel, then:
uint32_t v = MRT_MFC_FLAGS_DISABLE_WRONGVIF;
if (setsockopt(sock, IPPROTO_IP, MRT_API_CONFIG, (void *)&v, sizeof(v))
!= 0) {
return (ERROR);
}
if (v & MRT_MFC_FLAGS_DISABLE_WRONGVIF)
return (OK); /* Success */
else
return (ERROR);
In other words, when
setsockopt(
MRT_API_CONFIG) is called, the argument to it specifies the desired set of features to be enabled in the API and the kernel. The return value in
v is the actual (sub)set of features that were enabled in the kernel. To obtain later the same set of features that were enabled, then:
getsockopt(sock, IPPROTO_IP, MRT_API_CONFIG, (void *)&v, sizeof(v));
The set of enabled features is global. In other words,
setsockopt(
MRT_API_CONFIG) should be called right after
setsockopt(
MRT_INIT).
Currently, the following set of new features is defined:
#define MRT_MFC_FLAGS_DISABLE_WRONGVIF (1 << 0) /* disable WRONGVIF signals */
#define MRT_MFC_FLAGS_BORDER_VIF (1 << 1) /* border vif */
#define MRT_MFC_RP (1 << 8) /* enable RP address */
#define MRT_MFC_BW_UPCALL (1 << 9) /* enable bw upcalls */
The advanced multicast API uses a newly defined
struct mfcctl2 instead of the traditional
struct mfcctl. The original
struct mfcctl is kept as is. The new
struct mfcctl2 is:
/*
* The new argument structure for MRT_ADD_MFC and MRT_DEL_MFC overlays
* and extends the old struct mfcctl.
*/
struct mfcctl2 {
/* the mfcctl fields */
struct in_addr mfcc_origin; /* ip origin of mcasts */
struct in_addr mfcc_mcastgrp; /* multicast group associated*/
vifi_t mfcc_parent; /* incoming vif */
u_char mfcc_ttls[MAXVIFS];/* forwarding ttls on vifs */
/* extension fields */
uint8_t mfcc_flags[MAXVIFS];/* the MRT_MFC_FLAGS_* flags*/
struct in_addr mfcc_rp; /* the RP address */
};
The new fields are
mfcc_flags[MAXVIFS] and
mfcc_rp. Note that for compatibility reasons they are added at the end.
The
mfcc_flags[MAXVIFS] field is used to set various flags per interface per (S,G) entry. Currently, the defined flags are:
#define MRT_MFC_FLAGS_DISABLE_WRONGVIF (1 << 0) /* disable WRONGVIF signals */
#define MRT_MFC_FLAGS_BORDER_VIF (1 << 1) /* border vif */
The
MRT_MFC_FLAGS_DISABLE_WRONGVIF flag is used to explicitly disable the
IGMPMSG_WRONGVIF kernel signal at the (S,G) granularity if a multicast data packet arrives on the wrong interface. Usually, this signal is used to complete the shortest-path switch in case of PIM-SM multicast routing, or to trigger a PIM assert message. However, it should not be delivered for interfaces that are not in the outgoing interface set, and that are not expecting to become an incoming interface. Hence, if the
MRT_MFC_FLAGS_DISABLE_WRONGVIF flag is set for some of the interfaces, then a data packet that arrives on that interface for that MFC entry will NOT trigger a WRONGVIF signal. If that flag is not set, then a signal is triggered (the default action).
The
MRT_MFC_FLAGS_BORDER_VIF flag is used to specify whether the Border-bit in PIM Register messages should be set (in case when the Register encapsulation is performed inside the kernel). If it is set for the special PIM Register kernel virtual interface (see
pim(4)), the Border-bit in the Register messages sent to the RP will be set.
The remaining six bits are reserved for future usage.
The
mfcc_rp field is used to specify the RP address (in case of PIM-SM multicast routing) for a multicast group G if we want to perform kernel-level PIM Register encapsulation. The
mfcc_rp field is used only if the
MRT_MFC_RP advanced API flag/capability has been successfully set by
setsockopt(
MRT_API_CONFIG).
If the
MRT_MFC_RP flag was successfully set by
setsockopt(
MRT_API_CONFIG), then the kernel will attempt to perform the PIM Register encapsulation itself instead of sending the multicast data packets to user level (inside
IGMPMSG_WHOLEPKT upcalls) for user-level encapsulation. The RP address would be taken from the
mfcc_rp field inside the new
struct mfcctl2. However, even if the
MRT_MFC_RP flag was successfully set, if the
mfcc_rp field was set to
INADDR_ANY, then the kernel will still deliver an
IGMPMSG_WHOLEPKT upcall with the multicast data packet to the user-level process.
In addition, if the multicast data packet is too large to fit within a single IP packet after the PIM Register encapsulation (e.g., if its size was on the order of 65500 bytes), the data packet will be fragmented, and then each of the fragments will be encapsulated separately. Note that typically a multicast data packet can be that large only if it was originated locally from the same hosts that performs the encapsulation; otherwise the transmission of the multicast data packet over Ethernet for example would have fragmented it into much smaller pieces.
Typically, a multicast routing user-level process would need to know the forwarding bandwidth for some data flow. For example, the multicast routing process may want to timeout idle MFC entries, or in case of PIM-SM it can initiate (S,G) shortest-path switch if the bandwidth rate is above a threshold for example.
The original solution for measuring the bandwidth of a dataflow was that a user-level process would periodically query the kernel about the number of forwarded packets/bytes per (S,G), and then based on those numbers it would estimate whether a source has been idle, or whether the source's transmission bandwidth is above a threshold. That solution is far from being scalable, hence the need for a new mechanism for bandwidth monitoring.
Below is a description of the bandwidth monitoring mechanism.
•
If the bandwidth of a data flow satisfies some pre-defined filter, the kernel delivers an upcall on the multicast routing socket to the multicast routing process that has installed that filter.
•
The bandwidth-upcall filters are installed per (S,G). There can be more than one filter per (S,G).
•
Instead of supporting all possible comparison operations (i.e., < <= == != > >= ), there is support only for the <= and >= operations, because this makes the kernel-level implementation simpler, and because practically we need only those two. Further, the missing operations can be simulated by secondary user-level filtering of those <= and >= filters. For example, to simulate !=, then we need to install filter “bw <= 0xffffffff”, and after an upcall is received, we need to check whether “measured_bw != expected_bw”.
•
The bandwidth-upcall mechanism is enabled by setsockopt(MRT_API_CONFIG) for the MRT_MFC_BW_UPCALL flag.
•
The bandwidth-upcall filters are added/deleted by the new setsockopt(MRT_ADD_BW_UPCALL) and setsockopt(MRT_DEL_BW_UPCALL) respectively (with the appropriate struct bw_upcall argument of course).
From application point of view, a developer needs to know about the following:
/*
* Structure for installing or delivering an upcall if the
* measured bandwidth is above or below a threshold.
*
* User programs (e.g. daemons) may have a need to know when the
* bandwidth used by some data flow is above or below some threshold.
* This interface allows the userland to specify the threshold (in
* bytes and/or packets) and the measurement interval. Flows are
* all packet with the same source and destination IP address.
* At the moment the code is only used for multicast destinations
* but there is nothing that prevents its use for unicast.
*
* The measurement interval cannot be shorter than some Tmin (currently, 3s).
* The threshold is set in packets and/or bytes per_interval.
*
* Measurement works as follows:
*
* For >= measurements:
* The first packet marks the start of a measurement interval.
* During an interval we count packets and bytes, and when we
* pass the threshold we deliver an upcall and we are done.
* The first packet after the end of the interval resets the
* count and restarts the measurement.
*
* For <= measurement:
* We start a timer to fire at the end of the interval, and
* then for each incoming packet we count packets and bytes.
* When the timer fires, we compare the value with the threshold,
* schedule an upcall if we are below, and restart the measurement
* (reschedule timer and zero counters).
*/
struct bw_data {
struct timeval b_time;
uint64_t b_packets;
uint64_t b_bytes;
};
struct bw_upcall {
struct in_addr bu_src; /* source address */
struct in_addr bu_dst; /* destination address */
uint32_t bu_flags; /* misc flags (see below) */
#define BW_UPCALL_UNIT_PACKETS (1 << 0) /* threshold (in packets) */
#define BW_UPCALL_UNIT_BYTES (1 << 1) /* threshold (in bytes) */
#define BW_UPCALL_GEQ (1 << 2) /* upcall if bw >= threshold */
#define BW_UPCALL_LEQ (1 << 3) /* upcall if bw <= threshold */
#define BW_UPCALL_DELETE_ALL (1 << 4) /* delete all upcalls for s,d*/
struct bw_data bu_threshold; /* the bw threshold */
struct bw_data bu_measured; /* the measured bw */
};
/* max. number of upcalls to deliver together */
#define BW_UPCALLS_MAX 128
/* min. threshold time interval for bandwidth measurement */
#define BW_UPCALL_THRESHOLD_INTERVAL_MIN_SEC 3
#define BW_UPCALL_THRESHOLD_INTERVAL_MIN_USEC 0
The
bw_upcall structure is used as an argument to
setsockopt(
MRT_ADD_BW_UPCALL) and
setsockopt(
MRT_DEL_BW_UPCALL). Each
setsockopt(
MRT_ADD_BW_UPCALL) installs a filter in the kernel for the source and destination address in the
bw_upcall argument, and that filter will trigger an upcall according to the following pseudo-algorithm:
if (bw_upcall_oper IS ">=") {
if (((bw_upcall_unit & PACKETS == PACKETS) &&
(measured_packets >= threshold_packets)) ||
((bw_upcall_unit & BYTES == BYTES) &&
(measured_bytes >= threshold_bytes)))
SEND_UPCALL("measured bandwidth is >= threshold");
}
if (bw_upcall_oper IS "<=" && measured_interval >= threshold_interval) {
if (((bw_upcall_unit & PACKETS == PACKETS) &&
(measured_packets <= threshold_packets)) ||
((bw_upcall_unit & BYTES == BYTES) &&
(measured_bytes <= threshold_bytes)))
SEND_UPCALL("measured bandwidth is <= threshold");
}
In the same
bw_upcall the unit can be specified in both BYTES and PACKETS. However, the GEQ and LEQ flags are mutually exclusive.
Basically, an upcall is delivered if the measured bandwidth is >= or <= the threshold bandwidth (within the specified measurement interval). For practical reasons, the smallest value for the measurement interval is 3 seconds. If smaller values are allowed, then the bandwidth estimation may be less accurate, or the potentially very high frequency of the generated upcalls may introduce too much overhead. For the >= operation, the answer may be known before the end of
threshold_interval, therefore the upcall may be delivered earlier. For the <= operation however, we must wait until the threshold interval has expired to know the answer.
Example of usage:
struct bw_upcall bw_upcall;
/* Assign all bw_upcall fields as appropriate */
memset(&bw_upcall, 0, sizeof(bw_upcall));
memcpy(&bw_upcall.bu_src, &source, sizeof(bw_upcall.bu_src));
memcpy(&bw_upcall.bu_dst, &group, sizeof(bw_upcall.bu_dst));
bw_upcall.bu_threshold.b_data = threshold_interval;
bw_upcall.bu_threshold.b_packets = threshold_packets;
bw_upcall.bu_threshold.b_bytes = threshold_bytes;
if (is_threshold_in_packets)
bw_upcall.bu_flags |= BW_UPCALL_UNIT_PACKETS;
if (is_threshold_in_bytes)
bw_upcall.bu_flags |= BW_UPCALL_UNIT_BYTES;
do {
if (is_geq_upcall) {
bw_upcall.bu_flags |= BW_UPCALL_GEQ;
break;
}
if (is_leq_upcall) {
bw_upcall.bu_flags |= BW_UPCALL_LEQ;
break;
}
return (ERROR);
} while (0);
setsockopt(mrouter_s4, IPPROTO_IP, MRT_ADD_BW_UPCALL,
(void *)&bw_upcall, sizeof(bw_upcall));
To delete a single filter, then use
MRT_DEL_BW_UPCALL, and the fields of bw_upcall must be set exactly same as when
MRT_ADD_BW_UPCALL was called.
To delete all bandwidth filters for a given (S,G), then only the
bu_src and
bu_dst fields in
struct bw_upcall need to be set, and then just set only the
BW_UPCALL_DELETE_ALL flag inside field
bw_upcall.bu_flags.
The bandwidth upcalls are received by aggregating them in the new upcall message:
#define IGMPMSG_BW_UPCALL 4 /* BW monitoring upcall */
This message is an array of
struct bw_upcall elements (up to
BW_UPCALLS_MAX = 128). The upcalls are delivered when there are 128 pending upcalls, or when 1 second has expired since the previous upcall (whichever comes first). In an
struct upcall element, the
bu_measured field is filled-in to indicate the particular measured values. However, because of the way the particular intervals are measured, the user should be careful how
bu_measured.b_time is used. For example, if the filter is installed to trigger an upcall if the number of packets is >= 1, then
bu_measured may have a value of zero in the upcalls after the first one, because the measured interval for >= filters is “clocked” by the forwarded packets. Hence, this upcall mechanism should not be used for measuring the exact value of the bandwidth of the forwarded data. To measure the exact bandwidth, the user would need to get the forwarded packets statistics with the
ioctl(
SIOCGETSGCNT) mechanism (see the
Programming Guide section) .
Note that the upcalls for a filter are delivered until the specific filter is deleted, but no more frequently than once per
bu_threshold.b_time. For example, if the filter is specified to deliver a signal if bw >= 1 packet, the first packet will trigger a signal, but the next upcall will be triggered no earlier than
bu_threshold.b_time after the previous upcall.