ALTQ(9)

ALTQ(9) | Kernel Developer's Manual | ALTQ(9)

# NAME

`ALTQ` —
kernel interfaces for manipulating output queues on network interfaces

# SYNOPSIS

`#include <sys/types.h>`
`#include <sys/socket.h>`
`#include <net/if.h>`

void
`IFQ_ENQUEUE`(struct ifaltq *ifq, struct mbuf *m, struct altq_pktattr *pattr, int err);

void
`IFQ_DEQUEUE`(struct ifaltq *ifq, struct mbuf *m);

void
`IFQ_POLL`(struct ifaltq *ifq, struct mbuf *m);

void
`IFQ_PURGE`(struct ifaltq *ifq);

void
`IFQ_CLASSIFY`(struct ifaltq *ifq, struct mbuf *m, int af, struct altq_pktattr *pattr);

void
`IFQ_IS_EMPTY`(struct ifaltq *ifq);

void
`IFQ_SET_MAXLEN`(struct ifaltq *ifq, int len);

void
`IFQ_INC_LEN`(struct ifaltq *ifq);

void
`IFQ_DEC_LEN`(struct ifaltq *ifq);

void
`IFQ_INC_DROPS`(struct ifaltq *ifq);

void
`IFQ_SET_READY`(struct ifaltq *ifq);

# DESCRIPTION

The `ALTQ` system is a framework to manage queueing disciplines on network interfaces. `ALTQ` introduces new macros to manipulate output queues. The output queue macros are used to abstract queue operations and not to touch the internal fields of the output queue structure. The macros are independent from the `ALTQ` implementation, and compatible with the traditional `ifqueue` macros for ease of transition.

`IFQ_ENQUEUE`() enqueues a packet m to the queue ifq. The underlying queueing discipline may discard the packet. err is set to 0 on success, or `ENOBUFS` if the packet is discarded. m will be freed by the device driver on success or by the queueing discipline on failure, so that the caller should not touch m after calling `IFQ_ENQUEUE`().

`IFQ_DEQUEUE`() dequeues a packet from the queue. The dequeued packet is returned in m, or m is set to `NULL` if no packet is dequeued. The caller must always check m since a non-empty queue could return `NULL` under rate-limiting.

`IFQ_POLL`() returns the next packet without removing it from the queue. It is guaranteed by the underlying queueing discipline that `IFQ_DEQUEUE`() immediately after `IFQ_POLL`() returns the same packet.

`IFQ_PURGE`() discards all the packets in the queue. The purge operation is needed since a non-work conserving queue cannot be emptied by a dequeue loop.

`IFQ_CLASSIFY`() classifies a packet to a scheduling class, and returns the result in pattr.

`IFQ_IS_EMPTY`() can be used to check if the queue is empty. Note that `IFQ_DEQUEUE`() could still return `NULL` if the queueing discipline is non-work conserving.

`IFQ_SET_MAXLEN`() sets the queue length limit to the default FIFO queue.

`IFQ_INC_LEN`() and `IFQ_DEC_LEN`() increment or decrement the current queue length in packets.

`IFQ_INC_DROPS`() increments the drop counter and is equal to `IF_DROP`(). It is defined for naming consistency.

`IFQ_SET_READY`() sets a flag to indicate this driver is converted to use the new macros. `ALTQ` can be enabled only on interfaces with this flag.

# COMPATIBILITY

## ifaltq structure

In order to keep compatibility with the existing code, the new output queue structure `ifaltq` has the same fields. The traditional `IF_XXX`() macros and the code directly referencing the fields within `if_snd` still work with `ifaltq`. (Once we finish conversions of all the drivers, we no longer need these fields.)

                ##old-style##                           ##new-style##
                                           |
     struct ifqueue {                      | struct ifaltq {
        struct mbuf *ifq_head;             |    struct mbuf *ifq_head;
        struct mbuf *ifq_tail;             |    struct mbuf *ifq_tail;
        int          ifq_len;              |    int          ifq_len;
        int          ifq_maxlen;           |    int          ifq_maxlen;
        uint64_t     ifq_drops;            |    uint64_t     ifq_drops;
     };                                    |    /* altq related fields */
                                           |    ......
                                           | };
                                           |

The new structure replaces `struct ifqueue` in `struct ifnet`.

                ##old-style##                           ##new-style##
                                           |
     struct ifnet {                        | struct ifnet {
         ....                              |     ....
                                           |
         struct ifqueue if_snd;            |     struct ifaltq if_snd;
                                           |
         ....                              |     ....
     };                                    | };
                                           |

The (simplified) new `IFQ_XXX`() macros looks like:

    	#ifdef ALTQ
    	#define IFQ_DEQUEUE(ifq, m)			\
    		if (ALTQ_IS_ENABLED((ifq))		\
    			ALTQ_DEQUEUE((ifq), (m));	\
    		else					\
    			IF_DEQUEUE((ifq), (m));
    	#else
    	#define IFQ_DEQUEUE(ifq, m)	IF_DEQUEUE((ifq), (m));
    	#endif

## Enqueue operation

The semantics of the enqueue operation are changed. In the new style, enqueue and packet drop are combined since they cannot be easily separated in many queueing disciplines. The new enqueue operation corresponds to the following macro that is written with the old macros.

    #define	IFQ_ENQUEUE(ifq, m, pattr, err)                   \
    do {                                                      \
            if (ALTQ_IS_ENABLED((ifq)))                       \
                    ALTQ_ENQUEUE((ifq), (m), (pattr), (err)); \
            else {                                            \
                    if (IF_QFULL((ifq))) {                    \
                            m_freem((m));                     \
                            (err) = ENOBUFS;                  \
                    } else {                                  \
                            IF_ENQUEUE((ifq), (m));           \
                            (err) = 0;                        \
                    }                                         \
            }                                                 \
            if ((err))                                        \
                    (ifq)->ifq_drops++;                       \
    } while (false)

`IFQ_ENQUEUE`() does the following:

- queue a packet
- drop (and free) a packet if the enqueue operation fails

If the enqueue operation fails, err is set to `ENOBUFS`. m is freed by the queueing discipline. The caller should not touch mbuf after calling `IFQ_ENQUEUE`() so that the caller may need to copy m_pkthdr.len or m_flags field beforehand for statistics. The caller should not use `senderr`() since mbuf was already freed.

The new style `if_output`() looks as follows:

                ##old-style##                           ##new-style##
                                           |
     int                                   | int
     ether_output(ifp, m0, dst, rt0)       | ether_output(ifp, m0, dst, rt0)
     {                                     | {
         ......                            |     ......
                                           |
                                           |     mflags = m->m_flags;
                                           |     len = m->m_pkthdr.len;
         s = splimp();                     |     s = splimp();
         if (IF_QFULL(&ifp->if_snd)) {     |     IFQ_ENQUEUE(&ifp->if_snd, m,
                                           |                 NULL, error);
             IF_DROP(&ifp->if_snd);        |     if (error != 0) {
             splx(s);                      |         splx(s);
             senderr(ENOBUFS);             |         return (error);
         }                                 |     }
         IF_ENQUEUE(&ifp->if_snd, m);      |
         ifp->if_obytes +=                 |     ifp->if_obytes += len;
                        m->m_pkthdr.len;   |
         if (m->m_flags & M_MCAST)         |     if (mflags & M_MCAST)
             ifp->if_omcasts++;            |         ifp->if_omcasts++;
                                           |
         if ((ifp->if_flags & IFF_OACTIVE) |     if ((ifp->if_flags & IFF_OACTIVE)
             == 0)                         |         == 0)
             (*ifp->if_start)(ifp);        |         (*ifp->if_start)(ifp);
         splx(s);                          |     splx(s);
         return (error);                   |     return (error);
                                           |
     bad:                                  | bad:
         if (m)                            |     if (m)
             m_freem(m);                   |         m_freem(m);
         return (error);                   |     return (error);
     }                                     | }
                                           |

## Classifier

The classifier mechanism is currently implemented in `if_output`(). `struct altq_pktattr` is used to store the classifier result, and it is passed to the enqueue function. (We will change the method to tag the classifier result to mbuf in the future.)

    int
    ether_output(ifp, m0, dst, rt0)
    {
    	......
    	struct altq_pktattr pktattr;

    	......

    	/* classify the packet before prepending link-headers */
    	IFQ_CLASSIFY(&ifp->if_snd, m, dst->sa_family, &pktattr);

    	/* prepend link-level headers */
    	......

    	IFQ_ENQUEUE(&ifp->if_snd, m, &pktattr, error);

    	......
    }

# HOW TO CONVERT THE EXISTING DRIVERS

First, make sure the corresponding `if_output`() is already converted to the new style.

Look for if_snd in the driver. You will probably need to make changes to the lines that include if_snd.

## Empty check operation

If the code checks ifq_head to see whether the queue is empty or not, use `IFQ_IS_EMPTY`().

                ##old-style##                           ##new-style##
                                           |
     if (ifp->if_snd.ifq_head != NULL)     | if (IFQ_IS_EMPTY(&ifp->if_snd) == 0)
                                           |

Note that `IFQ_POLL`() can be used for the same purpose, but `IFQ_POLL`() could be costly for a complex scheduling algorithm since `IFQ_POLL`() needs to run the scheduling algorithm to select the next packet. On the other hand, `IFQ_IS_EMPTY`() checks only if there is any packet stored in the queue. Another difference is that even when `IFQ_IS_EMPTY`() is `false`, `IFQ_DEQUEUE`() could still return `NULL` if the queue is under rate-limiting.

## Dequeue operation

Replace `IF_DEQUEUE`() by `IFQ_DEQUEUE`(). Always check whether the dequeued mbuf is `NULL` or not. Note that even when `IFQ_IS_EMPTY`() is `false`, `IFQ_DEQUEUE`() could return `NULL` due to rate-limiting.

                ##old-style##                           ##new-style##
                                           |
     IF_DEQUEUE(&ifp->if_snd, m);          | IFQ_DEQUEUE(&ifp->if_snd, m);
                                           | if (m == NULL)
                                           |     return;
                                           |

A driver is supposed to call `if_start`() from transmission complete interrupts in order to trigger the next dequeue.

## Poll-and-dequeue operation

If the code polls the packet at the head of the queue and actually uses the packet before dequeueing it, use `IFQ_POLL`() and `IFQ_DEQUEUE`().

                ##old-style##                           ##new-style##
                                           |
     m = ifp->if_snd.ifq_head;             | IFQ_POLL(&ifp->if_snd, m);
     if (m != NULL) {                      | if (m != NULL) {
                                           |
         /* use m to get resources */      |     /* use m to get resources */
         if (something goes wrong)         |     if (something goes wrong)
             return;                       |         return;
                                           |
         IF_DEQUEUE(&ifp->if_snd, m);      |     IFQ_DEQUEUE(&ifp->if_snd, m);
                                           |
         /* kick the hardware */           |     /* kick the hardware */
     }                                     | }
                                           |

It is guaranteed that `IFQ_DEQUEUE`() immediately after `IFQ_POLL`() returns the same packet. Note that they need to be guarded by `splimp`() if called from outside of `if_start`().

## Eliminating IF_PREPEND

If the code uses `IF_PREPEND`(), you have to eliminate it since the prepend operation is not possible for many queueing disciplines. A common use of `IF_PREPEND`() is to cancel the previous dequeue operation. You have to convert the logic into poll-and-dequeue.

                ##old-style##                           ##new-style##
                                           |
     IF_DEQUEUE(&ifp->if_snd, m);          | IFQ_POLL(&ifp->if_snd, m);
     if (m != NULL) {                      | if (m != NULL) {
                                           |
         if (something_goes_wrong) {       |     if (something_goes_wrong) {
             IF_PREPEND(&ifp->if_snd, m);  |
             return;                       |         return;
         }                                 |     }
                                           |
                                           |     /* at this point, the driver
                                           |      * is committed to send this
                                           |      * packet.
                                           |      */
                                           |     IFQ_DEQUEUE(&ifp->if_snd, m);
                                           |
         /* kick the hardware */           |     /* kick the hardware */
     }                                     | }
                                           |

## Purge operation

Use `IFQ_PURGE`() to empty the queue. Note that a non-work conserving queue cannot be emptied by a dequeue loop.

                ##old-style##                           ##new-style##
                                           |
     while (ifp->if_snd.ifq_head != NULL) {|  IFQ_PURGE(&ifp->if_snd);
         IF_DEQUEUE(&ifp->if_snd, m);      |
         m_freem(m);                       |
     }                                     |
                                           |

## Attach routine

Use `IFQ_SET_MAXLEN`() to set ifq_maxlen to len. Add `IFQ_SET_READY`() to show this driver is converted to the new style. (This is used to distinguish new-style drivers.)

                ##old-style##                           ##new-style##
                                           |
     ifp->if_snd.ifq_maxlen = qsize;       | IFQ_SET_MAXLEN(&ifp->if_snd, qsize);
                                           | IFQ_SET_READY(&ifp->if_snd);
     if_attach(ifp);                       | if_attach(ifp);
                                           |

## Other issues

The new macros for statistics:

                ##old-style##                           ##new-style##
                                           |
     IF_DROP(&ifp->if_snd);                | IFQ_INC_DROPS(&ifp->if_snd);
                                           |
     ifp->if_snd.ifq_len++;                | IFQ_INC_LEN(&ifp->if_snd);
                                           |
     ifp->if_snd.ifq_len--;                | IFQ_DEC_LEN(&ifp->if_snd);
                                           |

Some drivers instruct the hardware to invoke transmission complete interrupts only when it thinks necessary. Rate-limiting breaks its assumption.

## How to convert drivers using multiple ifqueues

Some (pseudo) devices (such as slip) have another `ifqueue` to prioritize packets. It is possible to eliminate the second queue since `ALTQ` provides more flexible mechanisms but the following shows how to keep the original behavior.

    struct sl_softc {
    	struct	ifnet sc_if;		/* network-visible interface */
    	...
    	struct	ifqueue sc_fastq;	/* interactive output queue */
    	...
    };

The driver doesn't compile in the new model since it has the following line (if_snd is no longer a type of `struct ifqueue`).

    	struct ifqueue *ifq = &ifp->if_snd;

A simple way is to use the original `IF_XXX`() macros for sc_fastq and use the new `IFQ_XXX`() macros for if_snd. The enqueue operation looks like:

                ##old-style##                           ##new-style##
                                           |
     struct ifqueue *ifq = &ifp->if_snd;   | struct ifqueue *ifq = NULL;
                                           |
     if (ip->ip_tos & IPTOS_LOWDELAY)      | if ((ip->ip_tos & IPTOS_LOWDELAY) &&
         ifq = &sc->sc_fastq;              | !ALTQ_IS_ENABLED(&sc->sc_if.if_snd)) {
                                           |     ifq = &sc->sc_fastq;
     if (IF_QFULL(ifq)) {                  |     if (IF_QFULL(ifq)) {
         IF_DROP(ifq);                     |         IF_DROP(ifq);
         m_freem(m);                       |         m_freem(m);
         splx(s);                          |         error = ENOBUFS;
         sc->sc_if.if_oerrors++;           |     } else {
         return (ENOBUFS);                 |         IF_ENQUEUE(ifq, m);
     }                                     |         error = 0;
     IF_ENQUEUE(ifq, m);                   |     }
                                           | } else
                                           |     IFQ_ENQUEUE(&sc->sc_if.if_snd,
                                           |                 m, NULL, error);
                                           |
                                           | if (error) {
                                           |     splx(s);
                                           |     sc->sc_if.if_oerrors++;
                                           |     return (error);
                                           | }
     if ((sc->sc_oqlen =                   | if ((sc->sc_oqlen =
          sc->sc_ttyp->t_outq.c_cc) == 0)  |      sc->sc_ttyp->t_outq.c_cc) == 0)
         slstart(sc->sc_ttyp);             |     slstart(sc->sc_ttyp);
     splx(s);                              | splx(s);
                                           |

The dequeue operations looks like:

                ##old-style##                           ##new-style##
                                           |
     s = splimp();                         | s = splimp();
     IF_DEQUEUE(&sc->sc_fastq, m);         | IF_DEQUEUE(&sc->sc_fastq, m);
     if (m == NULL)                        | if (m == NULL)
         IF_DEQUEUE(&sc->sc_if.if_snd, m); |     IFQ_DEQUEUE(&sc->sc_if.if_snd, m);
     splx(s);                              | splx(s);
                                           |

# QUEUEING DISCIPLINES

Queueing disciplines need to maintain ifq_len (used by `IFQ_IS_EMPTY`()). Queueing disciplines also need to guarantee the same mbuf is returned if `IFQ_DEQUEUE`() is called immediately after `IFQ_POLL`().

# SEE ALSO

pf(4), altq.conf(5), pf.conf(5), altqd(8), tbrconfig(8)

# HISTORY

The `ALTQ` system first appeared in March 1997 and found its home in the KAME project (http://www.kame.net). It was imported into NetBSD 1.6.

October 24, 2022 | NetBSD 10.1