Chapter 10 — The Challenge Set

Answer: A

Explanation

Two logarithms with the same base add by multiplying what is inside them:

$$\log_2(x) + \log_2(x-2) = \log_2\big(x(x-2)\big) = 3$$

Undo the logarithm by writing it as a power of the base:

$$x(x - 2) = 2^3 = 8 \quad\Longrightarrow\quad x^2 - 2x - 8 = 0 \quad\Longrightarrow\quad (x - 4)(x + 2) = 0$$

So $x = 4$ or $x = -2$ — from the equation you rewrote, which is not quite the equation you were given.

Go back and test both. At $x = -2$ the original contains $\log_2(-2)$, and a logarithm of a negative number does not exist. That root is inadmissible. At $x = 4$: $\log_2 4 + \log_2 2 = 2 + 1 = 3$. ✓

So $x = 4$.

Why each wrong option is wrong:

• B, $-2$ — the forbidden root. It satisfies $x^2 - 2x - 8 = 0$ perfectly and cannot be substituted into the question. Combining two logarithms into one widens the domain, and everything gained that way has to be tested.
• C, $2$ — added the two roots. There is only one solution.
• D, $-8$ — multiplied them, giving the constant term of the quadratic.

Takeaway: Whenever you combine logarithms, you have changed the domain. Every root must go back into the original expression, and any that asks for the log of zero or a negative is discarded.

---

Answer: H

Explanation

There are two ways through, and they agree.

Directly. $2A = \begin{pmatrix} 4 & 2 \\ 6 & 8 \end{pmatrix}$, so

$$\det(2A) = 4 \times 8 - 2 \times 6 = 32 - 12 = 20$$

By the rule. Multiplying a matrix by a scalar multiplies every entry, and the determinant of a $2 \times 2$ is a product of two entries. So each term picks up two factors of the scalar:

$$\det(kA) = k^2 \det(A) \quad\text{for a } 2\times 2$$

Here $\det(A) = 2(4) - 1(3) = 5$, so $\det(2A) = 2^2 \times 5 = 20$.

Why each wrong option is wrong:

• F, $5$ — gave $\det(A)$ and ignored the scaling entirely.
• G, $10$ — multiplied the determinant by $2$ once. The scalar reaches every entry, not the determinant as a whole.
• J, $40$ — used $k^3$, which is the rule for a $3 \times 3$. The exponent is the size of the matrix.

Takeaway: $\det(kA) = k^n \det(A)$ for an $n \times n$ matrix. For the $2 \times 2$ matrices the ACT uses, that is $k^2$ — and if you cannot recall it, multiplying the four entries out takes ten seconds and settles it.

---

Answer: D

Explanation

A complex number cannot be left in a denominator. Clear it by multiplying top and bottom by the conjugate of the denominator — same real part, opposite imaginary part. For $1 - i$ that is $1 + i$.

$$\frac{2 + 3i}{1 - i} \times \frac{1 + i}{1 + i}$$

Top: $(2 + 3i)(1 + i) = 2 + 2i + 3i + 3i^2 = 2 + 5i - 3 = -1 + 5i$, using $i^2 = -1$.

Bottom: $(1 - i)(1 + i) = 1 - i^2 = 1 + 1 = 2$. The conjugate is chosen precisely so that this comes out real.

$$\frac{-1 + 5i}{2} = -\frac{1}{2} + \frac{5}{2}i$$

Why each wrong option is wrong:

• A, $-1 + 5 i$ — multiplied out the top correctly and never divided by the $2$ underneath. The last step is the one that gets dropped.
• B, $- \frac{1}{2} - \frac{5 i}{2}$ — right size, wrong sign on the imaginary part, from mishandling $3i^2$.
• C, $\frac{5}{2} + \frac{i}{2}$ — multiplied top and bottom by $1 - i$, the denominator itself rather than its conjugate. That leaves $(1-i)^2 = -2i$ underneath, still complex, which defeats the purpose.

Takeaway: Multiply by the conjugate, remember $i^2 = -1$, and split the result into real and imaginary halves at the end. If your denominator still has an $i$ in it, you used the wrong multiplier.

---

Answer: F

Explanation

Nothing is handed to you here — the first term, the last term and the count all have to be found before any formula can be used.

First multiple of 7 above 100: $7 \times 15 = 105$. Last multiple of 7 below 300: $7 \times 42 = 294$.

How many terms? They step by $7$, so

$$n = \frac{294 - 105}{7} + 1 = 27 + 1 = 28$$

The $+1$ is what counts the first term itself, and leaving it off is the classic fencepost error.

Now sum. An arithmetic series is the number of terms times the average of the first and last:

$$S = 28 \times \frac{105 + 294}{2} = 28 \times \frac{399}{2} = 14 \times 399 = 5586$$

Why each wrong option is wrong:

• G, $28$ — stopped at the number of terms.
• H, $399$ — gave $105 + 294$, the first plus the last, without averaging or multiplying.
• J, $8232$ — multiplied the count by the last term instead of by the average. That overcounts every term except the last.

Takeaway: For "the sum of all multiples of $k$ between two bounds", find the first and last multiples inside the range, count with $\tfrac{\text{last} - \text{first}}{k} + 1$, then use $S = n \times \tfrac{\text{first} + \text{last}}{2}$.

---

Answer: B

Explanation

Find the cosine first, then divide. The identity is

$$\sin^2\theta + \cos^2\theta = 1 \quad\Longrightarrow\quad \cos^2\theta = 1 - \left(\tfrac{3}{5}\right)^2 = \tfrac{16}{25} \quad\Longrightarrow\quad \cos\theta = \pm\tfrac{4}{5}$$

The sign is the question. In the second quadrant sine is positive and cosine is negative, so $\cos\theta = -\tfrac{4}{5}$.

$$\tan\theta = \frac{\sin\theta}{\cos\theta} = \frac{3/5}{-4/5} = -\frac{3}{4}$$

A quick sanity check: in the second quadrant only sine is positive, so tangent must be negative. Any positive option can be discarded before any arithmetic.

Why each wrong option is wrong:

• A, $\frac{3}{4}$ — took $\cos\theta = +\tfrac{4}{5}$ and ignored the range given. The range is the only reason it was stated.
• C, $- \frac{4}{3}$ — computed $\tfrac{\cos}{\sin}$, which is $\cot\theta$.
• D, $\frac{4}{3}$ — inverted and lost the sign.

Takeaway: The Pythagorean identity gives a magnitude; the quadrant gives the sign. Decide the sign before dividing, and check it against "all, sine, tan, cos" at the end.

---

Answer: J

Explanation

Add the vectors component by component first, then take one magnitude at the end. Magnitudes cannot be added — that is the whole difficulty.

$$\mathbf{u} + \mathbf{v} = \langle 5 + (-1),\; -12 + 5 \rangle = \langle 4, -7 \rangle$$

$$\|\langle 4, -7 \rangle\| = \sqrt{4^2 + (-7)^2} = \sqrt{16 + 49} = \sqrt{65}$$

That is about $8.06$.

Note that $\|\mathbf{u}\| = 13$ and $\|\mathbf{v}\| = \sqrt{26} \approx 5.1$, and the resultant is shorter than either sum you might have guessed — because the two vectors point in nearly opposite directions and partly cancel.

Why each wrong option is wrong:

• F, $11$ — added $4$ and $7$. That is the distance walked along the axes, not the straight-line length.
• G, $5 \sqrt{13}$ — subtracted the vectors instead of adding, giving $\langle 6, -17 \rangle$ and $\sqrt{325}$.
• H, $13$ — gave $\|\mathbf{u}\| = 13$ and never used $\mathbf{v}$.

Takeaway: Combine components, then apply Pythagoras once. $\|\mathbf{u} + \mathbf{v}\|$ is almost never $\|\mathbf{u}\| + \|\mathbf{v}\|$ — they are equal only when the vectors point the same way.

---

Answer: A

Explanation

A general-form equation tells you nothing until the squares are completed. Group the variables and move the constant across:

$$x^2 + 4x + y^2 - 6y = 12$$

$$(x + 2)^2 - 4 + (y - 3)^2 - 9 = 12$$

$$(x + 2)^2 + (y - 3)^2 = 12 + 4 + 9 = 25$$

The centre is $(-2, 3)$ and the radius is $\sqrt{25} = 5$.

The two corrections both add to the right-hand side, because both were subtracted on the left. Getting one of those signs wrong is what the other options are made of.

Why each wrong option is wrong:

• B, $2 \sqrt{3}$ — took the square root of the bare constant $12$, without adding what completing the squares contributed.
• C, $12$ — read the $12$ as $r^2$ and stopped.
• D, $25$ — reported $r^2 = 25$ rather than $r$. The last square root is the step most often skipped on this question.

Takeaway: Complete the square in both variables before reading anything off. Then $r = \sqrt{\text{right-hand side}}$ — and confirm you have taken the root, not left the square.

---

Answer: G

Explanation

"Without replacement" means the bag is different for the second draw, and both numbers in the fraction change.

First draw. $5$ red out of $8$ marbles: $\tfrac{5}{8}$.

Second draw. One red has gone, so $4$ red remain out of $7$ marbles: $\tfrac{4}{7}$.

The two must both happen, so multiply:

$$\frac{5}{8} \times \frac{4}{7} = \frac{20}{56} = \frac{5}{14}$$

That is about $0.357$.

Why each wrong option is wrong:

• F, $\frac{25}{64}$ — used $\tfrac{5}{8}$ twice, which is the answer with replacement.
• H, $\frac{5}{16}$ — reduced the number of reds to $4$ but left the total at $8$. Both counts drop, because the marble that left the numerator also left the bag.
• J, $\frac{15}{56}$ — computed red then blue.

Takeaway: Without replacement, decrease the top and the bottom for the second draw. Write the two fractions side by side before multiplying, and check that each denominator is one smaller than the last.

---

Answer: C

Explanation

You never have to build the inverse function. $f^{-1}(5)$ is, by definition, the input whose output is $5$ — so solve $f(x) = 5$.

$$\frac{2x + 1}{x - 3} = 5 \quad\Longrightarrow\quad 2x + 1 = 5(x - 3) = 5x - 15$$

$$16 = 3x \quad\Longrightarrow\quad x = \frac{16}{3}$$

Check: $f\!\left(\tfrac{16}{3}\right) = \tfrac{32/3 + 1}{16/3 - 3} = \tfrac{35/3}{7/3} = 5$. ✓

Why each wrong option is wrong:

• A, $3$ — gave $x = 3$, the value excluded from the domain because it makes the denominator zero.
• B, $\frac{11}{2}$ — computed $f(5) = \tfrac{11}{2}$. That is the function, not its inverse, and the two are only equal by coincidence.
• D, $- \frac{16}{3}$ — lost a sign moving the $-15$ across.

Takeaway: $f^{-1}(k)$ means "solve $f(x) = k$". Rearranging the whole function into an inverse formula is extra work and an extra chance to slip.

---

Answer: F

Explanation

Track the acid, and remember that adding pure acid increases the total volume as well. That second point is the whole question.

Acid at the start: $30\%$ of $20 = 6$ litres.

Add $L$ litres of pure acid. Acid becomes $6 + L$; total volume becomes $20 + L$.

The result must be $50\%$ acid:

$$6 + L = 0.5(20 + L) = 10 + 0.5L$$

$$0.5L = 4 \quad\Longrightarrow\quad L = 8$$

Check: $6 + 8 = 14$ litres of acid in $28$ litres of solution, and $\tfrac{14}{28} = 50\%$. ✓

Why each wrong option is wrong:

• G, $4$ — set the final amount against $50\%$ of the original $20$ litres, holding the total fixed. Pouring in $4$ litres would give $10$ litres of acid in $24$ litres of solution, about $42\%$.
• H, $20$ — assumed you must double the container to double the concentration. The concentration is not doubling; it is going from $30\%$ to $50\%$.
• J, $10$ — computed $50\%$ of $20$ and offered it as the amount to add.

Takeaway: In a mixture question, write "amount of the substance" on one side and "fraction × new total" on the other. Whatever you add goes into the total as well as the substance, unless the question says it evaporates.

---

Section 2 — Two Ideas at Once

The second ten join two pieces of content that the earlier chapters teach apart. A ratio question that is really about cubes. A logarithm that is really about which base you are standing in. A counting question where the objects are not all different. None of the individual ideas is new; the joining is the difficulty, and no line of the question tells you which two are being joined.

---

Answer: D

Explanation

An absolute value less than something is a band, and it opens into a double inequality:

$$|2x - 7| < 5 \quad\Longleftrightarrow\quad -5 < 2x - 7 < 5$$

Add $7$ throughout, then halve throughout:

$$2 < 2x < 12 \quad\Longrightarrow\quad 1 < x < 6$$

Now stop and read what that says. Both ends are strict, so $1$ and $6$ are out. The integers strictly between them are $2, 3, 4, 5$ — four of them.

Counting integers in an interval is where this goes wrong. Do not subtract the endpoints; list them if the range is short, and if it is long, count as "largest minus smallest, plus one" on the integers that are actually included — here $5 - 2 + 1 = 4$.

Why each wrong option is wrong:

• A, $5$ — computed $6 - 1$. That is the width of the interval, not the number of integers in it, and the two are never equal.
• B, $2$ — solved only $2x - 7 < 5$ with $2x - 7$ positive, giving $3.5 \le x < 6$ and just $\{4, 5\}$. An absolute value has two branches and both produce solutions.
• C, $6$ — read the sign as $\le$, which admits $x = 1$ and $x = 6$ and gives six integers. Check: $|2(1) - 7| = 5$, and $5$ is not less than $5$.

Takeaway: $|A| < b$ becomes $-b < A < b$; $|A| > b$ becomes two separate one-sided pieces. Solve to an interval, decide whether each end is included, then count the integers by listing or by "last minus first, plus one".

---

Answer: J

Explanation

Similar solids have one scale factor between them, and area and volume are different powers of it. Get the scale factor first; never scale one quantity by another quantity's ratio.

Length. The volumes are in the ratio $\tfrac{729}{216} = \tfrac{27}{8}$, and volume goes as the cube of length, so

$$\left(\frac{\text{length}_2}{\text{length}_1}\right)^3 = \frac{27}{8} \quad\Longrightarrow\quad \frac{\text{length}_2}{\text{length}_1} = \sqrt[3]{\frac{27}{8}} = \frac{3}{2}$$

Area. Area goes as the square of length:

$$240 \times \left(\frac{3}{2}\right)^2 = 240 \times \frac{9}{4} = 540$$

The whole family is worth carrying as one line: length $k$, area $k^2$, volume $k^3$. From any one of the three you can get to the other two.

Why each wrong option is wrong:

• F, $360$ — multiplied by $\tfrac{3}{2}$, the length ratio. Area is the square of it.
• G, $720$ — used $3$ as the scale factor, because $729 = 3^6$ and the cube root of $729$ is $9$… the reasoning varies, but the mistake is the same one: the ratio is $3$ to $2$, not $3$ to $1$. The smaller solid is not a unit.
• H, $810$ — multiplied by $\tfrac{27}{8}$, the volume ratio, which is the cube.

Takeaway: Find the length scale factor first, from whichever ratio you are given, then square it for area and cube it for volume. Scaling an area by a volume ratio is the single most common error on this question type.

---

Answer: A

Explanation

$\log_4 8$ asks: 4 to what power is 8? Neither $1$ nor $2$ works — $4^1 = 4$ and $4^2 = 16$ — so the answer is between them and is not a whole number. That alone rules out two options.

Write both numbers as powers of $2$ and let $y$ be the answer:

$$4^y = 8 \quad\Longrightarrow\quad (2^2)^y = 2^3 \quad\Longrightarrow\quad 2^{2y} = 2^3$$

Same base, so the exponents match: $2y = 3$ and $y = \tfrac{3}{2}$.

Change of base gets you there too, and is worth knowing because it works when the two numbers share no convenient base:

$$\log_4 8 = \frac{\log 8}{\log 4} = \frac{3\log 2}{2\log 2} = \frac{3}{2}$$

Why each wrong option is wrong:

• B, $3$ — computed $\log_2 8 = 3$, using base $2$ instead of base $4$. The small number is the base, and changing it changes the answer.
• C, $2$ — divided $8$ by $4$. A logarithm is an exponent, not a quotient.
• D, $\frac{2}{3}$ — computed $\log_8 4 = \tfrac{2}{3}$, with the base and the argument swapped. Notice that this is the reciprocal of the right answer, which is exactly what swapping them always produces.

Takeaway: Read $\log_b a$ as "$b$ to what power gives $a$", bracket the answer between two whole powers to sanity-check it, and use $\log_b a = \tfrac{\log a}{\log b}$ when the powers do not come out cleanly.

---

Answer: H

Explanation

Six letters in a row would give $6! = 720$ orders if every letter were different. They are not: BANANA is B, three A's and two N's.

Swapping two of the A's produces the same word, so the $720$ counts each real arrangement several times over — once for each way of shuffling the identical letters among themselves. There are $3!$ ways to permute the A's and $2!$ ways to permute the N's, so every distinguishable word has been counted $3! \times 2! = 6 \times 2 = 12$ times.

$$\frac{6!}{3! \, 2!} = \frac{720}{12} = 60$$

Why each wrong option is wrong:

• F, $720$ — $6! = 720$, treating all six letters as different.
• G, $120$ — divided by $3!$ for the A's and forgot the N's.
• J, $360$ — divided by $2!$ for the N's and forgot the A's.

Takeaway: For arrangements with repeats, divide the factorial of the total by the factorial of each repeated count. Every repeated letter needs its own factor in the denominator — count the letters, then count the divisions, and make sure the two lists agree.

---

Answer: B

Explanation

At an intersection the two graphs have the same $y$ for the same $x$, so set the right-hand sides equal:

$$x^2 - 3x + 2 = 2x - 4 \quad\Longrightarrow\quad x^2 - 5x + 6 = 0 \quad\Longrightarrow\quad (x - 2)(x - 3) = 0$$

So $x = 2$ and $x = 3$ — and those are $x$-coordinates. The question asked for $y$. Feed each one back into either equation; the line is easier:

$$y = 2(2) - 4 = 0 \qquad y = 2(3) - 4 = 2$$

The two points are $(2, 0)$ and $(3, 2)$, and the sum of the $y$-coordinates is $0 + 2 = 2$.

Why each wrong option is wrong:

• A, $5$ — $2 + 3 = 5$, the sum of the $x$-coordinates. It is also what the quadratic hands you directly as $-\tfrac{b}{a}$, which makes it very easy to reach and stop.
• C, $0$ — found $y = 0$ at the first point and stopped. Two intersections means two $y$-values.
• D, $6$ — $2 \times 3 = 6$, the product of the roots, and the constant term of $x^2 - 5x + 6$.

Takeaway: Solving a system gives you the $x$-values; substituting gives you the points. Before writing anything down, reread which coordinate — or which sum, or which difference — the question actually wants.

---

Answer: F

Explanation

Never divide both sides by $\sin\theta$. That is the whole question. Dividing by something that can be zero throws away every solution where it is zero, and here that is half of them.

Bring everything to one side and factor instead:

$$2\sin^2\theta - \sin\theta = 0 \quad\Longrightarrow\quad \sin\theta\,(2\sin\theta - 1) = 0$$

A product is zero when either factor is, so there are two equations to solve:

• $\sin\theta = 0$ gives $\theta = 0°$ and $\theta = 180°$ (not $360°$, which the range excludes)
• $\sin\theta = \tfrac{1}{2}$ gives $\theta = 30°$ and, because sine is positive in the second quadrant too, $\theta = 150°$

That is $0°, 30°, 150°, 180°$ — four values.

Why each wrong option is wrong:

• G, $3$ — kept both factors but took only the first-quadrant solution of $\sin\theta = \tfrac{1}{2}$, giving three.
• H, $1$ — did both: divided by $\sin\theta$ and then gave the single principal value $30°$.
• J, $2$ — divided through by $\sin\theta$, kept only $\sin\theta = \tfrac{1}{2}$, and found two.

Takeaway: Factor, never divide, when a trigonometric function appears on both sides. Then for each factor ask how many angles in one full turn give that value — usually two, and the CAST diagram says which two.

---

Answer: C

Explanation

Halve repeatedly and count the steps, then convert steps to years. Doing it by hand is fastest and least error-prone:

$$480 \to 240 \to 120 \to 60 \to 30 \to 15$$

That is five halvings. Each one takes $6$ years, so

$$5 \times 6 = 30 \text{ years}$$

Algebraically, with $n$ half-lives: $480 \times \left(\tfrac{1}{2}\right)^n = 15$, so $\left(\tfrac{1}{2}\right)^n = \tfrac{15}{480} = \tfrac{1}{32}$ and $n = 5$, because $2^5 = 32$.

Why each wrong option is wrong:

• A, $192$ — the mass fell by a factor of $\tfrac{480}{15} = 32$, and this treats $32$ as the number of half-lives: $32 \times 6 = 192$. The ratio is $32$; the number of halvings is the power of $2$ that makes $32$, which is $5$.
• B, $24$ — counted four halvings instead of five. The arrows are easy to miscount; the check is that the last number you write must be the target, $15$, not one step above it.
• D, $5$ — gave $5$, the number of half-lives, and never multiplied by the $6$ years each one takes.

Takeaway: Halving problems have two different numbers in them — how many times you halve, and how long each halving takes. Write the chain of masses out, count the arrows, then multiply.

---

Answer: G

Explanation

The midpoint formula averages the endpoints. Run it in reverse: if $M$ is the average of $A$ and $B$, then $B$ is twice $M$ minus $A$, coordinate by coordinate.

$$\frac{-2 + B_x}{2} = 3 \quad\Longrightarrow\quad B_x = 2(3) - (-2) = 8$$

$$\frac{4 + B_y}{2} = -1 \quad\Longrightarrow\quad B_y = 2(-1) - 4 = -6$$

So $B = (8, -6)$.

The step from $A$ to $M$ is $(+5, -5)$; the step from $M$ to $B$ must be the same again, landing on $(3 + 5, -1 - 5) = (8, -6)$. ✓ Walking the step twice is often quicker than the formula and self-checks as you go.

Why each wrong option is wrong:

• F, $\left( \frac{1}{2}, \ \frac{3}{2}\right)$ — averaged $M$ with $A$, giving the midpoint of $\overline{AM}$ instead of the far end of $\overline{AB}$.
• H, $\left( 5, \ -5\right)$ — gave $(5, -5)$, which is the step from $A$ to $M$, not a point. It is a useful quantity; it just needs adding to $M$.
• J, $\left( -7, \ 9\right)$ — computed $2A - M$, running the segment out from $A$ in the wrong direction. Check any candidate by averaging it with $A$: it must give $M$, and $(-7, 9)$ averages with $(-2, 4)$ to $(-4.5, 6.5)$.

Takeaway: $B = 2M - A$. Double the midpoint, subtract the known end, and verify by averaging your answer back with the known end.

---

Answer: D

Explanation

Means cannot be added or subtracted, but totals can. Convert both means to totals first — that is the only move this question needs.

Total of all $12$: $12 \times 45 = 540$. Total of the remaining $11$: $11 \times 43 = 473$.

The number removed is the difference between the two totals:

$$540 - 473 = 67$$

The size of it is worth noticing. The mean fell by only $2$, and the value removed was $67$ — well above the old mean of $45$. That is the point of the question: removing one number moves the mean of eleven others only a little, so the removed value has to be a long way out to do it.

Why each wrong option is wrong:

• A, $2$ — gave $45 - 43 = 2$, the change in the mean. Removing a $2$ from a list averaging $45$ would push the mean up, not down.
• B, $22$ — $11 \times 2 = 22$, multiplying the drop by the new count. Close in shape to a correct method, but the drop applies to eleven numbers while the removed value belongs to the original twelve.
• C, $24$ — $12 \times 2 = 24$, the same idea with the old count.

Takeaway: Turn every mean into a total the moment you see it: total $=$ mean $\times$ count. Once both totals are on the page, the answer is a subtraction, and a sanity check is free — the removed value must sit on the side of the old mean that explains which way the mean moved.

---

Answer: J

Explanation

Three sides and no angle, and no height either. Nothing you know about area applies until an angle exists, so make one with the law of cosines.

Step 1 — find $\cos B$. Angle $B$ sits between $AB$ and $BC$, and the side opposite it is $AC = 8$:

$$AC^2 = AB^2 + BC^2 - 2(AB)(BC)\cos B$$ $$64 = 25 + 49 - 2(5)(7)\cos B = 74 - 70\cos B$$ $$70\cos B = 10 \quad\Longrightarrow\quad \cos B = \frac{1}{7}$$

Step 2 — find $\sin B$. From $\sin^2 B + \cos^2 B = 1$, and taking the positive root because an angle in a triangle is between $0°$ and $180°$, where sine is always positive:

$$\sin B = \sqrt{1 - \tfrac{1}{49}} = \sqrt{\tfrac{48}{49}} = \frac{4\sqrt{3}}{7}$$

Step 3 — area. With two sides and the angle between them:

$$\text{Area} = \tfrac{1}{2}(AB)(BC)\sin B = \tfrac{1}{2}(5)(7)\cdot\frac{4\sqrt{3}}{7} = \tfrac{1}{2}(5)(4\sqrt{3}) = 10\sqrt{3}$$

That is about $17.3$ square units.

If you happen to know Heron's formula, it confirms this in one line: with $s = \tfrac{5+7+8}{2} = 10$,

$$\text{Area} = \sqrt{s(s-a)(s-b)(s-c)} = \sqrt{10 \cdot 5 \cdot 3 \cdot 2} = \sqrt{300} = 10\sqrt{3}$$

Why each wrong option is wrong:

• F, $\frac{35}{2}$ — $\tfrac{1}{2}(5)(7) = 17.5$, treating angle $B$ as a right angle. It is not: $5^2 + 7^2 = 74 \ne 64$. Notice how close $17.5$ is to the correct $17.3$ — the triangle is nearly right-angled, which is exactly why this option cannot be ruled out by estimating.
• G, $20$ — $\tfrac{1}{2}(5)(8) = 20$, using two sides as though one were a base and the other its height. A side is only a height when it is perpendicular to the base.
• H, $300$ — computed $s(s-a)(s-b)(s-c) = 10 \cdot 5 \cdot 3 \cdot 2 = 300$ under Heron's formula and never took the square root. A triangle with sides $5$, $7$ and $8$ cannot have an area of $300$; the largest area those sides could enclose is under $20$.

Takeaway: Three sides and no angle means the law of cosines. Get $\cos$ of the angle between the two sides you want to use, convert it to $\sin$ with the Pythagorean identity, then apply $\text{Area} = \tfrac{1}{2}ab\sin C$. Heron's formula is a shortcut worth knowing, and its square root is the step people drop.

---