Not sure if you are still looking for the solution but the ladder problem is pretty straight forward, it just gets a little messy. It's been about 7 years since I've taken geometry but all you need to solve this is the similar triangles theorem and some calculus to optimize for the shortest length.
If we were to draw out the problem, the shortest ladder would be the one which touches the top of the ladder, right? So this creates two similar triangles, one on the outside of the fence and the other above the fence but in between the fence and the wall. To clarify, in between these triangles there is a 5x5 square formed by the height of the fence and the distance from the fence to the wall. Let's call the distance from the fence to the bottom of the ladder 'a' and the length from the top of the ladder at the wall, to the top of the square, 'b'. Then by similar triangles, b / 5 = 5 / a or b = 25 / a. Using Pythagorus for the length of the ladder, L = sqrt((5+b)2 + (5+a)2). Into this expression we can plug in the expression for b from similar triangles. Since we want to optimize this problem to find the shortest length, it is easier to isolate the quantity under the root, and call it 'theta'. Expand theta, and then take the first derivative to get theta' = 2(a+5)(1 - 5(25)/a3). To find the minimum, set theta' = 0 and then it is clear that 1 = 125 / a3 or a = 5. Plug this back into the expression for L and that is your length. I got something like 14 feet.
The cylinder problem is also a simple optimization. The trick is to derive a relationship between the radius, 'r', and the height in the cone, 'h'. When r = 0, h = 5, and when r = 3.5, h = 0. Putting this in point slope form yields, h - 5 = [(5 - 0)/(0 - 3.5)](r - 0) or h = -(10/7)r + 5. We can plug this into the volume of the cylinder, V = pir2h. Take the first derivative wrt radius, set it equal to zero. This will give you the radius which gives the maximum volume. Plug this into the expression for h, and then plug both of these back into the volume for the cylinder and you will have your answer.
Hope that helps.
